Lexicographically sorting large files in Linux
- font size decrease font size increase font size
- Print Email
When I hear the word “sort” my first thought is usually “Hadoop”! Yes, sorting is one thing that Hadoop does well, but if you’re working with large files in Linux the built-in sort command is often all you need.
Let’s say you have a large file on a host with 2GB or more of main memory free. The following sort command is a efficient way to lexicographically-order large files.
LC_COLLATE=C sort --buffer-size=1G --temporary-directory=./tmp --unique bigfile.txt
Let’s break this command down and examine each part in detail.
( ! ) Warning: count(): Parameter must be an array or an object that implements Countable in /var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php on line 169 | ||||
---|---|---|---|---|
Call Stack | ||||
# | Time | Memory | Function | Location |
1 | 0.0018 | 412168 | {main}( ) | .../index.php:0 |
2 | 0.0838 | 4270368 | Joomla\CMS\Application\SiteApplication->execute( ) | .../index.php:49 |
3 | 0.0838 | 4270368 | Joomla\CMS\Application\SiteApplication->doExecute( ) | .../CMSApplication.php:196 |
4 | 0.2853 | 11392432 | Joomla\CMS\Application\SiteApplication->dispatch( ) | .../SiteApplication.php:233 |
5 | 0.2859 | 11416504 | Joomla\CMS\Component\ComponentHelper::renderComponent( ) | .../SiteApplication.php:194 |
6 | 0.2872 | 11471832 | Joomla\CMS\Component\ComponentHelper::executeComponent( ) | .../ComponentHelper.php:377 |
7 | 0.2875 | 11498800 | require_once( '/var/www/vhosts/shan.info/httpdocs/components/com_k2/k2.php' ) | .../ComponentHelper.php:402 |
8 | 0.2947 | 11895288 | K2ControllerItem->execute( ) | .../k2.php:64 |
9 | 0.2947 | 11895288 | K2ControllerItem->display( ) | .../BaseController.php:710 |
10 | 0.3032 | 12539696 | K2ControllerItem->display( ) | .../item.php:78 |
11 | 0.3032 | 12539696 | K2ControllerItem->display( ) | .../controller.php:19 |
12 | 0.3073 | 12906752 | Joomla\CMS\Cache\Controller\ViewController->get( ) | .../BaseController.php:663 |
13 | 0.3093 | 12927120 | K2ViewItem->display( ) | .../ViewController.php:102 |
14 | 0.3656 | 15732112 | K2ViewItem->display( ) | .../view.html.php:742 |
15 | 0.3656 | 15732112 | K2ViewItem->loadTemplate( ) | .../HtmlView.php:230 |
16 | 0.3672 | 15904416 | include( '/var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php' ) | .../HtmlView.php:701 |
- Published in CentOS 6
- Read 2433 times
( ! ) Notice: Only variables should be assigned by reference in /var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php on line 478 | ||||
---|---|---|---|---|
Call Stack | ||||
# | Time | Memory | Function | Location |
1 | 0.0018 | 412168 | {main}( ) | .../index.php:0 |
2 | 0.0838 | 4270368 | Joomla\CMS\Application\SiteApplication->execute( ) | .../index.php:49 |
3 | 0.0838 | 4270368 | Joomla\CMS\Application\SiteApplication->doExecute( ) | .../CMSApplication.php:196 |
4 | 0.2853 | 11392432 | Joomla\CMS\Application\SiteApplication->dispatch( ) | .../SiteApplication.php:233 |
5 | 0.2859 | 11416504 | Joomla\CMS\Component\ComponentHelper::renderComponent( ) | .../SiteApplication.php:194 |
6 | 0.2872 | 11471832 | Joomla\CMS\Component\ComponentHelper::executeComponent( ) | .../ComponentHelper.php:377 |
7 | 0.2875 | 11498800 | require_once( '/var/www/vhosts/shan.info/httpdocs/components/com_k2/k2.php' ) | .../ComponentHelper.php:402 |
8 | 0.2947 | 11895288 | K2ControllerItem->execute( ) | .../k2.php:64 |
9 | 0.2947 | 11895288 | K2ControllerItem->display( ) | .../BaseController.php:710 |
10 | 0.3032 | 12539696 | K2ControllerItem->display( ) | .../item.php:78 |
11 | 0.3032 | 12539696 | K2ControllerItem->display( ) | .../controller.php:19 |
12 | 0.3073 | 12906752 | Joomla\CMS\Cache\Controller\ViewController->get( ) | .../BaseController.php:663 |
13 | 0.3093 | 12927120 | K2ViewItem->display( ) | .../ViewController.php:102 |
14 | 0.3656 | 15732112 | K2ViewItem->display( ) | .../view.html.php:742 |
15 | 0.3656 | 15732112 | K2ViewItem->loadTemplate( ) | .../HtmlView.php:230 |
16 | 0.3672 | 15904416 | include( '/var/www/vhosts/shan.info/httpdocs/templates/gk_publisher/html/com_k2/templates/default/item.php' ) | .../HtmlView.php:701 |