Lexicographically sorting large files in Linux
- font size decrease font size increase font size
When I hear the word “sort” my first thought is usually “Hadoop”! Yes, sorting is one thing that Hadoop does well, but if you’re working with large files in Linux the built-in sort command is often all you need.
Let’s say you have a large file on a host with 2GB or more of main memory free. The following sort command is a efficient way to lexicographically-order large files.
LC_COLLATE=C sort --buffer-size=1G --temporary-directory=./tmp --unique bigfile.txtLet’s break this command down and examine each part in detail.
| ( ! ) Warning: count(): Parameter must be an array or an object that implements Countable in /var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php on line 169 | ||||
|---|---|---|---|---|
| Call Stack | ||||
| # | Time | Memory | Function | Location |
| 1 | 0.0009 | 415432 | {main}( ) | .../index.php:0 |
| 2 | 0.2267 | 4720640 | Joomla\CMS\Application\SiteApplication->execute( ) | .../index.php:49 |
| 3 | 0.2267 | 4720640 | Joomla\CMS\Application\SiteApplication->doExecute( ) | .../CMSApplication.php:196 |
| 4 | 0.6071 | 12448904 | Joomla\CMS\Application\SiteApplication->dispatch( ) | .../SiteApplication.php:233 |
| 5 | 0.6080 | 12473312 | Joomla\CMS\Component\ComponentHelper::renderComponent( ) | .../SiteApplication.php:194 |
| 6 | 0.6092 | 12528640 | Joomla\CMS\Component\ComponentHelper::executeComponent( ) | .../ComponentHelper.php:377 |
| 7 | 0.6095 | 12557464 | require_once( '/var/www/vhosts/shan.info/httpdocs/components/com_k2/k2.php' ) | .../ComponentHelper.php:402 |
| 8 | 0.6276 | 12992872 | K2ControllerItem->execute( ) | .../k2.php:64 |
| 9 | 0.6276 | 12992872 | K2ControllerItem->display( ) | .../BaseController.php:710 |
| 10 | 0.6540 | 13718072 | K2ControllerItem->display( ) | .../item.php:78 |
| 11 | 0.6540 | 13718072 | K2ControllerItem->display( ) | .../controller.php:19 |
| 12 | 0.6606 | 14114440 | Joomla\CMS\Cache\Controller\ViewController->get( ) | .../BaseController.php:663 |
| 13 | 0.6657 | 14134808 | K2ViewItem->display( ) | .../ViewController.php:102 |
| 14 | 0.7894 | 17106024 | K2ViewItem->display( ) | .../view.html.php:742 |
| 15 | 0.7894 | 17106024 | K2ViewItem->loadTemplate( ) | .../HtmlView.php:230 |
| 16 | 0.7926 | 17286224 | include( '/var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php' ) | .../HtmlView.php:701 |
- Published in CentOS 6
- Read 2438 times
| ( ! ) Notice: Only variables should be assigned by reference in /var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php on line 478 | ||||
|---|---|---|---|---|
| Call Stack | ||||
| # | Time | Memory | Function | Location |
| 1 | 0.0009 | 415432 | {main}( ) | .../index.php:0 |
| 2 | 0.2267 | 4720640 | Joomla\CMS\Application\SiteApplication->execute( ) | .../index.php:49 |
| 3 | 0.2267 | 4720640 | Joomla\CMS\Application\SiteApplication->doExecute( ) | .../CMSApplication.php:196 |
| 4 | 0.6071 | 12448904 | Joomla\CMS\Application\SiteApplication->dispatch( ) | .../SiteApplication.php:233 |
| 5 | 0.6080 | 12473312 | Joomla\CMS\Component\ComponentHelper::renderComponent( ) | .../SiteApplication.php:194 |
| 6 | 0.6092 | 12528640 | Joomla\CMS\Component\ComponentHelper::executeComponent( ) | .../ComponentHelper.php:377 |
| 7 | 0.6095 | 12557464 | require_once( '/var/www/vhosts/shan.info/httpdocs/components/com_k2/k2.php' ) | .../ComponentHelper.php:402 |
| 8 | 0.6276 | 12992872 | K2ControllerItem->execute( ) | .../k2.php:64 |
| 9 | 0.6276 | 12992872 | K2ControllerItem->display( ) | .../BaseController.php:710 |
| 10 | 0.6540 | 13718072 | K2ControllerItem->display( ) | .../item.php:78 |
| 11 | 0.6540 | 13718072 | K2ControllerItem->display( ) | .../controller.php:19 |
| 12 | 0.6606 | 14114440 | Joomla\CMS\Cache\Controller\ViewController->get( ) | .../BaseController.php:663 |
| 13 | 0.6657 | 14134808 | K2ViewItem->display( ) | .../ViewController.php:102 |
| 14 | 0.7894 | 17106024 | K2ViewItem->display( ) | .../view.html.php:742 |
| 15 | 0.7894 | 17106024 | K2ViewItem->loadTemplate( ) | .../HtmlView.php:230 |
| 16 | 0.7926 | 17286224 | include( '/var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php' ) | .../HtmlView.php:701 |