Indexes are taking up extra space on disk

CrateDB version: 2.1.5

JVM version: OpenJDK 1.8.0_151

OS version / environment description: Ubuntu / AWS / Local SSD Raid-0

Problem description: Indexes end up using almost 2x the disk space they require over time. I never experienced this with older versions - I ran on 0.xx.x releases for nearly 2 years without issue. With newer releases - now on 2.1.5 - I have problems with disks filling up on nodes. Over time it ends up using 2x the index size 😕

Right now, I wiped out one of the shard replicas and am letting it rebuild:

posts        0  p STARTED      6691734  54.9gb 10.113.185.220 Crève Tête
posts        0  r INITIALIZING                 10.63.224.7    Cime de Bolofré```

After it finished:

posts        0  p STARTED 6692042  54.9gb 10.113.185.220 Crève Tête
posts        0  r STARTED 6692041  35.6gb 10.63.224.7    Cime de Bolofré

~20GB of dead weight sitting on each node. This is costly because we are using local SSD-based nodes and are limited in available storage space.

Steps to reproduce:

Posts Schema:

create table posts (
    id string primary key,
    domain_id string,
    feed_id string,
    url string,
    title string INDEX using fulltext with(analyzer='standard'),
    by_line string INDEX using fulltext,
    description string INDEX using fulltext with(analyzer='standard'),
    image_url string INDEX OFF,
    content string INDEX using fulltext with(analyzer='standard'),
    links array(object as (url string, anchor string)),
    category string,
    word_count integer,
    tags string,
    language string,
    types array(string),
    author_clearvoice_ids array(string),
    googleplus_source string,
    googleplus_type string,
    author_googleplus_id string,
    twitter_source string,
    twitter_type string,
    author_twitter_id string,
    document_filename string,
    document_start_position integer INDEX OFF,
    document_end_position integer INDEX OFF,
    social_fb_comments integer,
    social_fb_likes integer,
    social_fb_shares integer,
    social_gpluses integer,
    social_linkedin integer,
    social_pinterest integer,
    social_tweets integer,
    social_total integer,
    social_updates array(object as (updated_at timestamp, tweets integer, facebook_comments integer, facebook_likes integer, facebook_shares integer, linkedin_shares integer, google_plus_ones integer, pinterest_pins integer, total integer)),
    created_at timestamp,
    updated_at timestamp,
    published_at timestamp,
    social_updated_at timestamp,
    social_scheduled_at timestamp
) clustered into 12 shards with (number_of_replicas=1)

Sample Records:

[
  {
    "author_clearvoice_ids": null, 
    "author_googleplus_id": null, 
    "author_twitter_id": null, 
    "by_line": "Michael Savage", 
    "category": null, 
    "content": "When the most senior officials across Whitehall gathered for a rare away day on Friday, an attack on the impartiality of the civil service by a leading Brexiter ensured they had plenty to talk about between the presentations and flow charts.\n\nSince the referendum there have been a string of allegations thrown at the politicians and institutions perceived as failing in their duty to deliver a clean break from the European Union. However, while pro-Remain MPs have largely been the target to date, the civil service is increasingly in the crosshairs.\n\nLast week\u2019s row erupted after Jacob Rees-Mogg, the leader of the pro-Brexit European Research Group of Tory MPs, suggested in the Commons that Treasury officials had deliberately drawn up economic models designed to undermine Brexit. Steve Baker, the influential pro-Leave minister, apologised for initially failing to challenge the claim \u2013 which made David Davis, the Brexit secretary, visibly wince.\n\nEven before the spat, some senior Whitehall figures had already noticed a new tendency to lay the failures of the government\u2019s Brexit policy at the civil service\u2019s door. Just hours before Rees-Mogg made his allegations, Gus O\u2019Donnell, the former cabinet secretary, was down the corridor in the Lords, telling peers that his former Whitehall colleagues were under attack. \u201cIt is particularly disappointing to see a divided cabinet resorting to attacking the civil servants who simply want to implement whatever policy cabinet finally decides upon,\u201d he warned. \u201cThe bickering and blaming of civil servants needs to stop.\u201d\n\nThe claim in today\u2019s Observer by former cabinet secretary Andrew Turnbull that leading Brexiters have adopted similar tactics to the \u201cstab in the back\u201d myth \u2013 propagated in Germany after the first world war to suggest its military had been betrayed by internal forces \u2013 is the most incendiary description of Brexiter tactics. However, others have raised concerns about the willingness by some to attack pillars of Britain\u2019s constitutional make-up \u2013 be it judges, politicians or officials. Andrew Cooper, formerly David Cameron\u2019s pollster and a key figure in the Remain campaign, said Rees-Mogg\u2019s attack showed that \u201chard-Brexiteers are the UK manifestation of bullying post-Truth Trumpite alt-right\u201d. \u201cObjective analysis is a conspiracy, actual facts met with alternative facts; considering risks and costs is treason, reporting them is fake news,\u201d he said.\n\nNick Macpherson, a former chief civil servant at the Treasury, tweeted yesterday: \u201cFirst it was the socialists, then the unions, the immigrants and Brussels bureaucrats. Now it\u2019s the treacherous Treasury. #fantasyisland.\u201d\n\nMany Whitehall insiders are prepared for even more heated rhetoric as Britain is forced to make Brexit trade-offs. Some opted not to speak out this week, keeping their powder dry for \u201cwhen it goes really wrong\u201d.\n\nWith cabinet ministers preparing for a two-day crunch meeting this week over the Brexit deal, some officials believe the attacks on Whitehall are actually a proxy, designed to remind those gathered to deliver a meaningful Brexit.\n\nMeanwhile, Rees-Mogg, Tory grassroots favourite for leadership, yesterday stood by his claim that in drawing up its Brexit plans, the Treasury was trying to influence the argument. He also said that Charles Grant, head of the Centre for European Reform, had been given improper briefings. \u201cHe is getting private briefings from the Treasury against government policy,\u201d he told BBC Radio 4\u2019s Today. \u201cThis is very serious. It is not for officials to invent policy.\n\n\u201cWith the referendum and with the EU, the Treasury has gone back to making forecasts. It was politically advantageous in the past. It is the same for them now. I do think they are fiddling the figures.\u201d\n\nGrant said it was perfectly normal for officials to talk to think tanks. \u201cThe Treasury cares about economics so it is naturally pushing for the sorts of Brexit that minimise the economic damage,\u201d he said.", 
    "created_at": 1517694627329, 
    "description": "Row erupts after Jacob Rees-Mogg suggested in the Commons that Whitehall was deliberately undermining leave process", 
    "document_end_position": 7854902, 
    "document_filename": "2018/2/3/21/1517694367_383218", 
    "document_start_position": 7682757, 
    "domain_id": "13012715934034554483", 
    "feed_id": "17557947423604253280", 
    "googleplus_source": null, 
    "googleplus_type": null, 
    "id": "16112374660569751432", 
    "image_url": "https://i.guim.co.uk/img/media/52b164256d91d280a797224a1dfed33fe2c6e1c3/0_146_2861_1716/master/2861.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&crop=faces%2Centropy&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxOC8wMS8zMS9mYWNlYm9va19kZWZhdWx0LnBuZw&s=29004454eb2c3e2c997b7484154ea5cb", 
    "language": null, 
    "links": null, 
    "published_at": 1517693408000, 
    "social_fb_comments": null, 
    "social_fb_likes": null, 
    "social_fb_shares": 2700, 
    "social_gpluses": 0, 
    "social_linkedin": 14, 
    "social_pinterest": 93, 
    "social_scheduled_at": 1517990746150, 
    "social_total": 2807, 
    "social_tweets": null, 
    "social_updated_at": 1517875214018, 
    "social_updates": [
      {
        "linkedin_shares": 0, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517719836000, 
        "facebook_shares": 96, 
        "total": 127
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517734242000, 
        "facebook_shares": 171, 
        "total": 203
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517741457000, 
        "facebook_shares": 778, 
        "total": 811
      }, 
      {
        "linkedin_shares": 4, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517745072000, 
        "facebook_shares": 1100, 
        "total": 1135
      }, 
      {
        "linkedin_shares": 5, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517748675000, 
        "facebook_shares": 1400, 
        "total": 1436
      }, 
      {
        "linkedin_shares": 6, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517752282000, 
        "facebook_shares": 1600, 
        "total": 1637
      }, 
      {
        "linkedin_shares": 6, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517755896000, 
        "facebook_shares": 1700, 
        "total": 1737
      }, 
      {
        "linkedin_shares": 6, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517763134000, 
        "facebook_shares": 1900, 
        "total": 1937
      }, 
      {
        "linkedin_shares": 6, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517766764000, 
        "facebook_shares": 2000, 
        "total": 2037
      }, 
      {
        "linkedin_shares": 7, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517774031000, 
        "facebook_shares": 2200, 
        "total": 2238
      }, 
      {
        "linkedin_shares": 8, 
        "pinterest_pins": 62, 
        "google_plus_ones": 0, 
        "updated_at": 1517788568000, 
        "facebook_shares": 2400, 
        "total": 2470
      }, 
      {
        "linkedin_shares": 9, 
        "pinterest_pins": 62, 
        "google_plus_ones": 0, 
        "updated_at": 1517817447000, 
        "facebook_shares": 2500, 
        "total": 2571
      }, 
      {
        "linkedin_shares": 14, 
        "pinterest_pins": 93, 
        "google_plus_ones": 0, 
        "updated_at": 1517875214000, 
        "facebook_shares": 2700, 
        "total": 2807
      }
    ], 
    "tags": null, 
    "title": "\u2018Treacherous Treasury\u2019 claims signal a new target for hard-Brexiters", 
    "twitter_source": null, 
    "twitter_type": null, 
    "types": [
      "article"
    ], 
    "updated_at": 1517694627324, 
    "url": "https://www.theguardian.com/politics/2018/feb/03/treasury-target-for-hard-brexiters-jacob-rees-mogg", 
    "word_count": 625
  }, 
  {
    "author_clearvoice_ids": null, 
    "author_googleplus_id": null, 
    "author_twitter_id": "14010511297414611016", 
    "by_line": "Harriet Sherwood", 
    "category": null, 
    "content": "Its gothic twin turrets and stained-glass window featuring a six-pointed star look out from a hillside over a town in south Wales. A Welsh dragon decorates the building\u2019s gable. But rooms that once resonated to the murmur of prayers and readings from the Torah are abandoned; windows are broken, plaster is crumbling and the roof is open to the sky.\n\nBut now the Old Synagogue in Merthyr Tydfil, built in the 1870s, could be reborn. It is part of an extraordinary scheme \u2013 to be launched this week by the historian Simon Schama \u2013 to map more than 3,300 historic synagogues across 48 European countries, and restore the most significant sites.\n\nThe synagogue in Merthyr Tydfil was the centre of a community of around 400 Jews, many from eastern Europe. Its members ran a button factory, a chocolate business, a betting shop, property companies and other local enterprises. The annual Jewish Ball was attended by many of the town\u2019s citizens, Jews and non-Jews.\n\nBut by the 1980s, a minyan \u2013 a quorum of 10 men \u2013 could no longer be reached, and the synagogue was sold. The grade II-listed building became a Christian centre and later a gym; today it lies empty and vandalised.\n\nNow, however, there is hope that it will be preserved and restored as a Jewish museum, part of the scheme being rolled out this week. The project, commissioned by the Foundation for Jewish Heritage, has identified synagogues built before the second world war, from Cork in Ireland in the west to Vladivostok in Russia in the east. Each has been catalogued with construction dates and materials, the Jewish community it served, its present use and condition, and a \u201csignificance rating\u201d.\n\nSchama will launch the project in parliament on Wednesday with the backing of more than 40 high-profile supporters including Downton Abbey creator Julian Fellowes, architect Daniel Libeskind, television newsreader Natasha Kaplinsky, artist Anish Kapoor, authors Linda Grant and Howard Jacobson, and former government ministers Malcolm Rifkind and Tristram Hunt.\n\nBefore 1939, there were an estimated 17,000 synagogues across Europe, but the majority have been lost. Of the 3,318 surviving buildings, only 718 still function as Jewish places of worship; others are abandoned, in ruins or used for other purposes such as warehousing, factories, restaurants and theatres. One houses a swimming pool; others are funeral homes or fire stations.\n\nThe project faced \u201cspecial challenges around Jewish heritage\u201d, especially in eastern Europe, said foundation member Michael Mail. \u201cThe Holocaust was followed by communism. Many buildings were abandoned and essentially lost their communities of users. In preserving these buildings, we also preserve the stories of the communities that for hundreds of years were the heartlands of the Jewish people. These places can serve as profound portals into the worlds that were once there.\u201d\n\nThe inventory was undertaken by the Center for Jewish Art at the Hebrew University of Jerusalem, assisted by heritage experts in individual countries.\n\n\u201cWe can\u2019t save them all, so let\u2019s save the best, the most important, the most at risk,\u201d said Mail. \u201cWe\u2019ve homed in on 160, and narrowed those down to 19 that we\u2019re particularly looking at, where there\u2019s a good chance of saving and restoring the buildings. Each one of those has a different story. In many cases, these buildings are the last witnesses to a Jewish life that was. This is not just Jewish heritage: it is Europe\u2019s cultural and historical heritage and we\u2019re in a race against time to save it.\u201d\n\nOne of the first buildings in line for restoration is the Great Synagogue in Slonim, Belarus, built in the 1640s. Before the second world war, 17,000 Jews lived in Slonim, more than two-thirds of the local population. An estimated 200 survived.\n\nThe synagogue, a baroque building overlooking the marketplace, was used as a warehouse after the war but has been abandoned for 18 years. It has been vandalised and is in danger of collapse, but some of the interior paintings and carvings are intact.\n\nAmong those rounded up and killed in Slonim were Kaplinksy\u2019s relatives. The newsreader discovered her Jewish family history when she travelled to the city for the television series Who Do You Think You Are?\n\n\u201cIt was devastating to find out that a large number of my family were killed by the Nazis,\u201d she told the Observer. \u201cOne key moment [in the Slonim trip] was going to the synagogue where most of my family used to worship before being rounded up and burned alive.\u201d\n\nThe synagogue is a \u201cmajestic building, absolutely stunning. You can see its history on its walls, but it is falling apart. I was horrified to find swastikas painted on the outside walls.\u201d\n\nAfter the programme was made, 27 Kaplinsky family members from all over the world met in Belarus to learn more about their history. \u201cWe ended up in the synagogue,\u201d Kaplinsky said. \u201cIt was hugely symbolic that the building that tore our family apart brought us back together. It was a very special moment.\u201d\n\nThere were lessons to be learned from the past, she added. \u201cWhen you look around the world you can see the devastation caused by prejudice and hatred. We need to educate future generations and remind them of history.\u201d\n\nDiscussions are now under way to restore the Slonim synagogue as a Jewish museum, educational and cultural centre, and a place of worship.\n\nIn Merthyr Tydfil, the proposal to restore the Old Synagogue as a Jewish museum of Wales and cultural centre is supported by the city council and local politicians.\n\nGerald Jones, the Labour MP for Merthyr Tydfil and Rhymney, said the restoration plan \u201cwould see this building once again playing a part in the life of our community\u201d. The foundation is seeking funding from partners including the Heritage Lottery Fund in the UK. It is also hoping for donations from people with family connections to synagogues.", 
    "created_at": 1517692575247, 
    "description": "Simon Schama, Howard Jacobson and Natasha Kaplinsky among supporters of campaign to protect Jewish heritage", 
    "document_end_position": 7547883, 
    "document_filename": "2018/2/3/22/1517695744_332971", 
    "document_start_position": 7377778, 
    "domain_id": "13012715934034554483", 
    "feed_id": "853803068455874689", 
    "googleplus_source": null, 
    "googleplus_type": null, 
    "id": "4211104451318915241", 
    "image_url": "https://i.guim.co.uk/img/media/ec9a8a6d570df14d6d8f49c6fbb20cabac23f126/0_226_4272_2563/master/4272.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&crop=faces%2Centropy&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxOC8wMS8zMS9mYWNlYm9va19kZWZhdWx0LnBuZw&s=fc316ff038640dd7ba57d05fa6d9b0e2", 
    "language": null, 
    "links": null, 
    "published_at": 1517691608000, 
    "social_fb_comments": null, 
    "social_fb_likes": null, 
    "social_fb_shares": 1200, 
    "social_gpluses": 0, 
    "social_linkedin": 57, 
    "social_pinterest": 94, 
    "social_scheduled_at": 1517977851513, 
    "social_total": 1351, 
    "social_tweets": null, 
    "social_updated_at": 1517927272044, 
    "social_updates": [
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517699716000, 
        "facebook_shares": 85, 
        "total": 117
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517714123000, 
        "facebook_shares": 141, 
        "total": 173
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517721333000, 
        "facebook_shares": 175, 
        "total": 207
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517724944000, 
        "facebook_shares": 196, 
        "total": 228
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517728549000, 
        "facebook_shares": 232, 
        "total": 264
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517732166000, 
        "facebook_shares": 266, 
        "total": 298
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517735771000, 
        "facebook_shares": 316, 
        "total": 349
      }, 
      {
        "linkedin_shares": 1, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517739382000, 
        "facebook_shares": 356, 
        "total": 389
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517742990000, 
        "facebook_shares": 380, 
        "total": 414
      }, 
      {
        "linkedin_shares": 8, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517750212000, 
        "facebook_shares": 454, 
        "total": 494
      }, 
      {
        "linkedin_shares": 12, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517753828000, 
        "facebook_shares": 485, 
        "total": 529
      }, 
      {
        "linkedin_shares": 16, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517761074000, 
        "facebook_shares": 542, 
        "total": 590
      }, 
      {
        "linkedin_shares": 16, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517764706000, 
        "facebook_shares": 568, 
        "total": 616
      }, 
      {
        "linkedin_shares": 21, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517771974000, 
        "facebook_shares": 688, 
        "total": 741
      }, 
      {
        "linkedin_shares": 22, 
        "pinterest_pins": 32, 
        "google_plus_ones": 0, 
        "updated_at": 1517775620000, 
        "facebook_shares": 735, 
        "total": 789
      }, 
      {
        "linkedin_shares": 30, 
        "pinterest_pins": 63, 
        "google_plus_ones": 0, 
        "updated_at": 1517826147000, 
        "facebook_shares": 1000, 
        "total": 1093
      }, 
      {
        "linkedin_shares": 40, 
        "pinterest_pins": 63, 
        "google_plus_ones": 0, 
        "updated_at": 1517851421000, 
        "facebook_shares": 1000, 
        "total": 1103
      }, 
      {
        "linkedin_shares": 52, 
        "pinterest_pins": 94, 
        "google_plus_ones": 0, 
        "updated_at": 1517901982000, 
        "facebook_shares": 1100, 
        "total": 1246
      }, 
      {
        "linkedin_shares": 57, 
        "pinterest_pins": 94, 
        "google_plus_ones": 0, 
        "updated_at": 1517927272000, 
        "facebook_shares": 1200, 
        "total": 1351
      }
    ], 
    "tags": null, 
    "title": "Plan to save Europe\u2019s synagogues receives high-profile backing", 
    "twitter_source": "post", 
    "twitter_type": "semantic", 
    "types": [
      "article"
    ], 
    "updated_at": 1517696110877, 
    "url": "https://www.theguardian.com/world/2018/feb/03/save-synagogues-europe-jewish-heritage-simon-schama", 
    "word_count": 977
  }, 
  {
    "author_clearvoice_ids": null, 
    "author_googleplus_id": null, 
    "author_twitter_id": "14084104844907538833", 
    "by_line": "Ben Quinn", 
    "category": null, 
    "content": "A spectre is haunting Europe in 2018 \u2013 to borrow from one of his catchier one-liners \u2013 the spectre of Karl Marx himself.\n\nTwo hundred years after the philosopher\u2019s birth, a small industry is gathering pace, from plans for major events in Trier, the city on the Moselle where he was born, to a new tour of the Manchester streets that he and Friedrich Engels walked as they discussed the condition of the city\u2019s emerging working class. The bicentenary on 5 May will be marked with exhibitions, lectures, conferences, histories and novels.\n\nThe books are starting to pile up. Last month saw a new edition of Marxism \u2013 a Graphic Guide, a collaboration by philosophy lecturer Rupert Woodfin and comic book artist Oscar Z\u00e1rate, while titles by heavyweight specialists on Marxism are on the way. They include a reprint of literary theorist Terry Eagleton\u2019s bestselling Why Marx Was Right, along with a new edition of The Communist Manifesto \u2013 which starts with the \u201cspectre\u201d quotation \u2013 including an introduction by the former Greek finance minister Yanis Varoufakis.\n\nMarx\u2019s ideas, running through the Russian revolution to the present day, will be the focus of Marx and Marxism, a new book by one of Britain\u2019s foremost historians of socialism, Gregory Claeys. The influence of the Labour party under Jeremy Corbyn \u2013 as well as factors such as reduced employment prospects and a desire to challenge austerity \u2013 are credited by Claeys as helping to engender a renewed interest in Marx, particularly among the young.\n\n\u201cMarx\u2019s prose may seem somewhat obtuse to modern readers,\u201d Claeys said. \u201cBut Marx\u2019s central premise \u2013 that the most obvious and extreme forms of oppression and exploitation can be removed from everyday life \u2013 retains a robustness and daring paralleled by no other thinkers in the modern period.\u201d\n\nFact is accompanied by fiction. The Murder of Warren Street by Oxford university historian Marc Mulholland, published at the end of May, promises to tell the story of villain Emmanuel Barth\u00e9lemy (\u201cthe man who wanted to kill Marx\u201d).\n\nMarx Returns, due out on 23 February and written by Jason Barker, is billed as combining historical fiction, psychological mystery, philosophy and extracts from Marx\u2019s and Engels\u2019s collected works to reimagine the life and times of Marx.\n\nAmong a plethora of gatherings and conferences being organised by the various families of the left, one of the most eagerly awaited is Marx 200, a major conference due to take place at Soas University of London and organised by the Marx Memorial Library.\n\nShadow chancellor John McDonnell \u2013 arguably Britain\u2019s best-known Marxist \u2013 will speak on the theme of \u201cInto the 21st century: Marxism as a force for change today\u201d alongside guests from around the world, including Sitaram Yechury, the general secretary of the Communist party of India (Marxist), and Luo Wendong, a professor from the Chinese Academy of Social Sciences.\n\nMeirian Jump, archivist at the memorial library, said interest had increased in Marxism in the past couple of years, while numbers attending lectures on Marxism and conducting research in the library\u2019s reading room have risen in recent months.\n\n\u201cIn the autumn our venue reached capacity and we had to turn people away from our lectures celebrating 150 years since the publication of Das Kapital,\u201d Jump said. \u201cIt was noticeable that a large number of those queuing outside Marx House were young people and students.\u201d\n\nAway from the political calls to arms or Marxist think-ins, exhibitions include the Karl and Eleanor Marx Treasures Gallery, from May to early August at the British Library. The display aims to explore the role that the British Museum reading room, a predecessor institution of the British Library, played in the life and work of Marx and his daughter, a writer and political activist in her own right.\n\nItems on display will include correspondence by Marx, his family and Friedrich Engels, covering both personal and political affairs, as well as rare copies of first editions of Marx\u2019s writings, several of which he donated to the library. Among these is a copy of the first French translation of Das Kapital, believed to feature annotations in Marx\u2019s own hand.\n\nTo the likely chagrin of committed Marxists and eurosceptics, the distinctly un-Marxist figure of European Commission president Jean-Claude Juncker will open a series of exhibitions in Trier. Visitors will be able to view a new permanent exhibition at the Karl-Marx-Haus Museum, and a bronze figure of Marx donated by China.\n\nThose unable to make the trip might instead consider the Marx 200th birthday walking tour in Manchester, where Engels lived on and off for almost 30 years and was visited by Marx.\n\n\u201cWe\u2019ve been doing Marx-themed walks for a while. He and Engels were great drinkers so we did one based on the pubs they used to go to, and there was a great response,\u201d said Ed Glinert of New Manchester Walks.\n\n\u201cYou get a real range of people. I took the Chinese consul around one time, for example. We don\u2019t get too many Americans, though.\u201d\n\nAs for what Marx would make of it all, Claeys asserted he had \u201ca fine, robust sense of humour\u201d and would certainly have mocked many who have taken up his name over the past 150 years.\n\n\u201cHe would, I think, be a \u2018deep green\u2019 thinker who would advocate sustainable development, an end to planned obsolescence and production based on the profit rather than global human need,\u201d he said.", 
    "created_at": 1517692574954, 
    "description": "Renewed interest in philosopher fires celebrations of 200 years since his birth on 5 May 1818", 
    "document_end_position": 6691125, 
    "document_filename": "2018/2/4/10/1517741293_306383", 
    "document_start_position": 6514839, 
    "domain_id": "13012715934034554483", 
    "feed_id": "11216890166714227096", 
    "googleplus_source": null, 
    "googleplus_type": null, 
    "id": "10842689211995424991", 
    "image_url": "https://i.guim.co.uk/img/media/5d7503100e9c79646a25955dffa590c802a5b68c/0_363_5616_3370/master/5616.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&crop=faces%2Centropy&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxOC8wMS8zMS9mYWNlYm9va19kZWZhdWx0LnBuZw&s=b1d08d9682d3a44e55264179af22dd71", 
    "language": null, 
    "links": [
      {
        "url": "https://www.eventbrite.co.uk/e/marx-200-tickets-41952397751", 
        "anchor": "major conference"
      }, 
      {
        "url": "http://www.newmanchesterwalks.com/walks-tours/political-manchester/engels-marx-the-frock-coated-communists-2/", 
        "anchor": "Marx 200th birthday"
      }
    ], 
    "published_at": 1517699040000, 
    "social_fb_comments": null, 
    "social_fb_likes": null, 
    "social_fb_shares": 1300, 
    "social_gpluses": 0, 
    "social_linkedin": 2, 
    "social_pinterest": 94, 
    "social_scheduled_at": 1517955550038, 
    "social_total": 1396, 
    "social_tweets": null, 
    "social_updated_at": 1517912290496, 
    "social_updates": [
      {
        "linkedin_shares": 0, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517696180000, 
        "facebook_shares": 51, 
        "total": 82
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 63, 
        "google_plus_ones": 0, 
        "updated_at": 1517782592000, 
        "facebook_shares": 974, 
        "total": 1039
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 63, 
        "google_plus_ones": 0, 
        "updated_at": 1517825810000, 
        "facebook_shares": 1200, 
        "total": 1265
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 63, 
        "google_plus_ones": 0, 
        "updated_at": 1517847424000, 
        "facebook_shares": 1200, 
        "total": 1265
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 94, 
        "google_plus_ones": 0, 
        "updated_at": 1517890660000, 
        "facebook_shares": 1300, 
        "total": 1396
      }, 
      {
        "linkedin_shares": 2, 
        "pinterest_pins": 94, 
        "google_plus_ones": 0, 
        "updated_at": 1517912290000, 
        "facebook_shares": 1300, 
        "total": 1396
      }
    ], 
    "tags": null, 
    "title": "Marx bicentenary to be marked by exhibitions, books \u2013 and pub crawls", 
    "twitter_source": "post", 
    "twitter_type": "semantic", 
    "types": [
      "article"
    ], 
    "updated_at": 1517741863149, 
    "url": "https://www.theguardian.com/books/2018/feb/03/karl-marx-bicentenary-events-books-exhibitions", 
    "word_count": 902
  }, 
  {
    "author_clearvoice_ids": null, 
    "author_googleplus_id": null, 
    "author_twitter_id": null, 
    "by_line": "Robin McKie", 
    "category": null, 
    "content": "illian Hastings Ward gave birth to her second child, Sam, almost four years ago. For the first few months of his life, the boy appeared to be in good health. \u201cThen we realised that he was not making proper visual contact, and discovered he was blind,\u201d Hastings Ward recalls. Subsequent diagnosis also revealed that Sam was not progressing intellectually. \u201cHis brain just couldn\u2019t join the dots,\u201d she says. Today Sam has the mental development of a six-month-old child.\n\nAt the time of his diagnosis, it was not apparent what was affecting him. Then Hastings Ward and her husband Nick, who live in Bristol, were told about a pioneering scheme launched by the Department of Health. The 100,000 Genomes Project involves several teams of scientists, all of whom have been working towards a remarkably ambitious goal: the sequencing of 100,000 genomes of individuals affected by rare disorders and cancers. Later this month, the project \u2013 which was given the go-ahead by prime minister David Cameron in 2012 \u2013 will reveal that it has reached its halfway point and has sequenced its 50,000th genome. This news will be followed with the announcement later this year of major initiatives aimed at ensuring that the UK becomes a world leader in genomics medicine.\n\nThe implications are enormous. Major steps are being taken towards the creation of a health service in which healing is dovetailed \u2013 or personalised \u2013 to suit the needs of individual patients. Instead of taking a one-size-fits-all approach, treatments will be tailored to fit the makeup of individual patients.\n\nThe story of Sam Ward reveals the enormous potential of this approach to healthcare. After being enrolled in the 100,000 Genomes Project, Sam\u2019s DNA was sampled along with that of his mother and his father. Each of their genomes \u2013 their entire complement of genes \u2013 were then sequenced.\n\nGiven that each of us has a genome made up of more than 3 billion letters of DNA, this decoding represents an enormous task, one that has required the use of some of the world\u2019s most advanced sequencing machines as well as computers capable of storing\u00a0millions of gigabytes of data. Last year several thousand genomes were sequenced this way. Those of the Ward family were among them.\n\n\u201cIt was then that we found out what was affecting Sam,\u201d said Hastings Ward, a former civil servant. The boy had a fault in the gene Grin-1, a very rare mutation that causes moderate to severe intellectual disability, low muscle tone and, in some cases, seizures. Knowing what was affecting their son was, on its own, a considerable relief. There were other \u2013 even more welcome \u2013 benefits, however.\n\nBy studying the genomes of Sam\u2019s parents, doctors were able to show that neither had passed on the Grin-1 gene variant to their son. It had arisen, by chance, as a mutation inside his own DNA. \u201cThat was tremendously important,\u201d says Hastings Ward. \u201cIt showed that it is extremely unlikely that his elder sister Kirsty would be affected by the condition. That had been a real worry\u00a0for us.\u201d\n\nIn addition, there has been the development \u2013 by researchers in the US \u2013 of a possible drug that could treat the condition. \u201cThat has been another very positive development for us. The project has brought tremendous relief and hope,\u201d says Hastings Ward, who now heads a panel that represents the interests of the volunteer participants in the project.\n\nDisorders such as the one caused by Grin-1 are uncommon but they still represent a significant cumulative burden on the health service.\n\n\u201cThere are between 7,000 and 8,000 rare disorders like the one that affects Sam,\u201d says Professor Mark Caulfield, the project\u2019s chief scientist.\n\n\u201cSome do not have names and in many cases we do not know the causes. And yes, each is rare: but because there are so many of them they still have a considerable impact on the wellbeing of the nation. Almost 3\u00a0million people are affected by a rare disorder in the UK. Usually these manifest themselves at an early age, in the first two or three years of life, and about a third of those affected will die by the age of five.\u201d\n\nThis grim statistic explains why rare disorders were pinpointed by the founders of the project and why early successes are providing encouraging hopes for future developments in diagnoses and treatments. Hastings Ward says: \u201cWhen you have an undiagnosed child, you have no idea what lies ahead for your family. Thanks to this project, we have now found some clues, a network of families who are in a similar situation, and more hope for the future.\u201d\n\nThis point is backed by Juliet Mills, who lives in Worcestershire and whose 13-year-old son Gabriel suffers from the rare disorder nemaline myopathy \u2013 a condition that leaves him struggling to walk and requiring a wheelchair to make longer journeys. \u201cSix or seven gene mutations have been linked to the disease but Gabriel does not have any of these,\u201d she says. \u201cWe have joined the 100,000 Genomes Project and are awaiting results. If we can find out exactly what is causing his condition, that will give us some hope that one day some form of treatment could be developed to help him.\u201d\n\nRare disorders are not the only target, however. Another major component is the study and diagnosis of cancer. Patients with particular tumours are being recruited and their genomes sequenced. The genomes of their tumours are also being sequenced to trace the molecular change that turned a normally healthy piece of DNA into one that makes cells divide uncontrollably. Of the 50,000 genomes that have been sequenced so far, around 8,000 have been targeted for cancer research. These are also showing encouraging results.\n\n\u201cIf I developed cancer today \u2013 from what I have now seen \u2013 I would want to have my tumour genome sequenced,\u201d says Caulfield. \u201cThese sequences are showing us how to think about tailoring therapies and offer clinical trial opportunities that could have enormous potential. We are highlighting what changes in DNA have occurred in a patient and, crucially, we are also able to provide links to potential clinical trial opportunities from which a clinician can make a selection that best suits their patient.\n\n\u201cWe are starting to use the molecular signature of a cancer to direct a therapy for a particular patient and, hopefully, provide some with better, longer lives. This could convert some cancers to simple, longer-term\u00a0diseases.\u201d\n\nThis enthusiasm is shared by Professor Sue Hill, chief scientific officer for England says: \u201cI have never seen a transformation project that has achieved quite so much in such a short time,\u201d she says. \u201cThese technologies are going to be embedded in the NHS through the new genomic medicine service from this autumn.\u201d\n\nThis point is emphasised by Caulfield. \u201cFrom October, this technology is going to become available not just to those taking part in our project but to all those families who are touched by rare diseases and by certain cancers. We cannot yet say which cancers, but the long-term aim is to include all that respond to this\u00a0approach.\u201d\n\nGeneticists point out that the cost of genome sequencing is also expected to decline sharply and become even more affordable. A full genome sequence currently costs around \u00a31,000, and many less efficient diagnostic tests could be replaced. \u201cThat will release money to provide further funds for genomic medicine,\u201d says Caulfield. \u201cThe benefits of this technology are going to be profound.\u201d\n\nAlex Masterson\n\nOver the first 18 years of his life, Alex Masterson has had 28 operations, including the removal of tumours and several bouts of heart surgery. Doctors originally thought he was suffering from a condition known as Noonan syndrome, but doubts persisted. His symptoms \u2013 which included facial tumours \u2013 did not quite fit this diagnosis. These doubts were unsettling for his family.\n\nTwo years ago his mother, Kirsty, was told about the 100,000 Genomes Project by the family\u2019s genetic counsellor and enrolled Alex. Sequencing of his genome revealed he had a related condition known as Leopard syndrome. \u201cIt has not changed his life expectancy or anything like that. However, it has given us closure and that has been a marvellous relief,\u201d she says.\n\nDiagnoses like Alex\u2019s can also bring alleviation from the odysseys of diagnostic visits that families with rare disorders have to go through. An affected child can face hundreds of outpatient visits to different specialists in the first few years of life as doctors struggle to trace the causes of their symptoms. By pinpointing a condition\u2019s cause, the 100,000 Genomes Project could bring an end to the stress, discomfort and costs involved, say researchers.\n\nDilys Neill\n\nDilys Neill was a doctor who specialised in the treatment of childhood cancers when, in October 2016, she was diagnosed as having acute myeloid leukaemia, a condition in which a person\u2019s bone marrow produces white blood cells that grow and divide too fast. These abnormal cells then build up in the blood and bone marrow.\n\n\u201cMy first thought was that I would be dead before our daughter\u2019s wedding the next year and that her day would be spoiled,\u201d she recalls.\n\nHowever, a transplant of stem cells from her sister Ros gave Dr Neill a lifeline \u2013 to the extent that she will celebrate her 64th birthday on Monday and has been told that she has a 50-50 chance of surviving the next five\u00a0years.\n\n\u201cAs a prognosis, it could be better, but it could be a lot worse,\u201d says Dr Neill, who has now retired and lives with her husband, William, in Stow-on-the-Wold, Gloucestershire.\n\n\u201cHowever, it would be better for future generations if we had ways to pinpoint the condition before it has had a chance to take a hold in a patient,\u201d she adds. And that is why she has joined the 100,000 Genomes\u00a0Project.\n\n\u201cIf a lot of patients with my condition have their genomes sequenced, it may be possible to pinpoint particular stretches of DNA that predispose certain individuals to acute myeloid leukaemia. That would be an enormous step forward in designing new treatments for the condition.\u201d\n\nJim Wright\n\nJim Wright is 17 and a wheelchair user. His mother Karen had noticed he had problems walking when he was a toddler. \u201cHe just couldn\u2019t climb stairs,\u201d she says. Jim was eventually diagnosed as having Charcot-Marie-Tooth disease, a genetic condition appearing either in childhood or later in life and characterised by loss of muscle strength in the legs. However, the condition comes in numerous varieties and doctors could not determine which variant affected Jim.\n\nWhen the family heard about the 100,000 Genomes Project, they asked to be enrolled. Samples of DNA were taken and it was discovered that Jim had a version of Charcot-Marie-Tooth that is known as type 2z and which is caused by an alteration in his MORC2 gene. \u201cMore importantly, we found out that neither my husband nor I carried the gene responsible for our son\u2019s condition. It was a new mutation that had first appeared in Jim,\u201d says\u00a0Karen.\n\nFor the Wrights \u2013 who live in Cannock, Staffordshire \u2013 this was a welcome diagnosis. \u201cIt means that our other son, Sam, is not likely to have picked up a gene for the condition and so he does not have to worry about fathering children who might inherit the condition. It also means that Jim could also father healthy children in the future \u2013 by using pre-implantation diagnosis to pinpoint embryos that do not carry the gene. This has all brought tremendous relief for my husband Kevin and I.\u201d\n\nAnd Jim is equally excited. \u201cIt\u2019s good to be part of a project that could bring better diagnoses and treatments for future generations,\u201d he says.\n\nThe prime goal of the 100,000 Genomes Project is to sequence the entire complement of genes possessed by around 70,000 individuals in the UK.\n\n\u201cIn some cases, we will sequence a person\u2019s genome and nothing else,\u201d says Professor Mark Caulfield, the project\u2019s chief scientist. \u201cIn other cases, usually in patients affected by cancers, we will sequence their genomes and also the genomes of their tumours. And in others we will sequence the genomes of the affected person as well as the genomes of their parents.\u201d\n\nResearchers are already close to the halfway stage of their endeavour. \u201cIn the next few weeks, we expect to notch up our 50,000th genome,\u201d says Caulfield. Given that it took separate teams of US and UK geneticists almost a decade to unravel the very first human genome, in 2003, at a cost of around $3bn, this progress is remarkable.\n\nConstant improvements in sequencing technology and in data storage have played a key role in this dramatic jump in performance.\n\n\u201cEvery three years, a new generation of sequencing machines is developed and so costs and sequencing rates have been slashed,\u201d Caulfield says. \u201cOur project will achieve its goal on a budget of \u00a3300m provided entirely by the government. Without these improvements in sequencing technology, it would have cost around \u00a3200tn to fund the project.\u201d\n\nTo sequence genomes, blood samples from patients and family members are sent to centres round England (Scotland, North Ireland and Wales have their own partnership projects) and then collected at a repository in Milton Keynes. From there, the samples are sent to the NHS Genome Sequencing Centre at the Sanger Institute south of Cambridge, where every one of the 3.1bn letters of DNA in each genome will be read. Around 21m gigabytes of data are expected to be generated and will be stored at the project\u2019s data centre in Corsham, Wiltshire, where it will be used to help diagnose patients\u2019 conditions and provide data for scientists studying cancers and inherited diseases.", 
    "created_at": 1517691089711, 
    "description": "The future now holds hope for children like Sam Ward, and doctors say many other people are set to benefit", 
    "document_end_position": 5288137, 
    "document_filename": "2018/2/3/20/1517690791_658756", 
    "document_start_position": 5105893, 
    "domain_id": "13012715934034554483", 
    "feed_id": "2830464843753963620", 
    "googleplus_source": null, 
    "googleplus_type": null, 
    "id": "15583413505091406265", 
    "image_url": "https://i.guim.co.uk/img/media/65bcef3a12aa435ee8497f2a2d592a1e3bf32583/0_108_2592_1555/master/2592.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&crop=faces%2Centropy&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxOC8wMS8zMS9mYWNlYm9va19kZWZhdWx0LnBuZw&s=e8be6ee2ceab11059bfb712dbc28e280", 
    "language": null, 
    "links": [
      {
        "url": "https://www.genomicsengland.co.uk/the-100000-genomes-project/", 
        "anchor": "100,000 Genomes Project"
      }
    ], 
    "published_at": 1517689807000, 
    "social_fb_comments": null, 
    "social_fb_likes": null, 
    "social_fb_shares": 938, 
    "social_gpluses": 0, 
    "social_linkedin": 129, 
    "social_pinterest": 93, 
    "social_scheduled_at": 1518008207766, 
    "social_total": 1160, 
    "social_tweets": null, 
    "social_updated_at": 1517921664556, 
    "social_updates": [
      {
        "linkedin_shares": 4, 
        "pinterest_pins": 31, 
        "google_plus_ones": 0, 
        "updated_at": 1517694703000, 
        "facebook_shares": 47, 
        "total": 82
      }, 
      {
        "linkedin_shares": 49, 
        "pinterest_pins": 62, 
        "google_plus_ones": 0, 
        "updated_at": 1517781116000, 
        "facebook_shares": 613, 
        "total": 724
      }, 
      {
        "linkedin_shares": 66, 
        "pinterest_pins": 62, 
        "google_plus_ones": 0, 
        "updated_at": 1517824331000, 
        "facebook_shares": 740, 
        "total": 868
      }, 
      {
        "linkedin_shares": 88, 
        "pinterest_pins": 62, 
        "google_plus_ones": 0, 
        "updated_at": 1517845948000, 
        "facebook_shares": 834, 
        "total": 984
      }, 
      {
        "linkedin_shares": 101, 
        "pinterest_pins": 62, 
        "google_plus_ones": 0, 
        "updated_at": 1517856762000, 
        "facebook_shares": 861, 
        "total": 1024
      }, 
      {
        "linkedin_shares": 119, 
        "pinterest_pins": 93, 
        "google_plus_ones": 0, 
        "updated_at": 1517878392000, 
        "facebook_shares": 903, 
        "total": 1115
      }, 
      {
        "linkedin_shares": 129, 
        "pinterest_pins": 93, 
        "google_plus_ones": 0, 
        "updated_at": 1517921664000, 
        "facebook_shares": 938, 
        "total": 1160
      }
    ], 
    "tags": null, 
    "title": "The DNA database that is key to beating our rarest diseases", 
    "twitter_source": null, 
    "twitter_type": null, 
    "types": [
      "article"
    ], 
    "updated_at": 1517691089706, 
    "url": "https://www.theguardian.com/science/2018/feb/03/1000-genomes-project-genetics-dna-database-key-beating-rarest-disorders", 
    "word_count": 2288
  }, 
  {
    "author_clearvoice_ids": null, 
    "author_googleplus_id": null, 
    "author_twitter_id": null, 
    "by_line": "Paul MacInnes", 
    "category": null, 
    "content": "Two goals of a quality befitting a higher division gave Wolves victory over Sheffield United as the hosts stamped their authority on yet another Championship promotion contender.\n\nA long-range beauty from R\u00faben Neves and a team move finished by Diogo Jota gave Wolves a comfortable half-time lead. The dismissal of the United goalkeeper, Simon Moore, with 15 minutes to go, followed by a third goal from the resultant free-kick, merely rubbed salt into the visitors\u2019 wounds. They were game but the Blades were thoroughly outclassed.\n\nThere was a big-match atmosphere inside Molineux led by a raucous Sir Jack Hayward stand and Wolves immediately rose to the occasion. In the fifth minute Conor Coady played a pass from centre-half that found Ivan Cavaleiro on the left wing. He cut inside, forcing George Baldock into a blocking challenge, and the loose ball rolled to a waiting Neves. The Portugal midfielder promptly wrapped his cultured right boot around it and curled a perfect shot into the top right corner of the net.\n\nIt was a goal of real quality and yet on the half-hour, Wolves trumped it. Again the move involved Cavaleiro, the winger driving on the ball through the centre of the field before playing it to the right. A short series of passes ended with Matt Doherty picking his cross, low to Jota on the penalty spot. The forward feigned to shoot, instead hit a cheeky short pass to Cavaleiro, advanced on to the return ball and tucked home his 12th league goal of the season. It was a bit tasty.\n\n\u201cIt was a good performance, a good result and a good crowd,\u201d said the Wolves manager, Nuno Esp\u00edrito Santo. \u201cThe boys enjoyed the game. We were able to control them, play football. It\u2019s a good result and a good crowd. The second goal put us ahead in the game. It was important to achieve that, the production of the team needed a goal to finish the game and that\u2019s why I was happy about it.\u201d\n\nUnited had kicked off in a 3-5-2 formation, with the 30-year-old Ricky Holmes \u2013 making his debut at this level after his move from Charlton \u2013 playing in the hole. At half\u2011time, Chris Wilder shook up his formation to 4-4-2, with Chris Basham stepping into midfield and Holmes pulling wide. It brought them back into the game in terms of possession but also back into the familiar position of pushing at Wolves without seeing any rewards for their endeavour.\n\nWilder said it was his intention to stay in the match but with a quarter of an hour remaining, it slipped away from United. Diogo, chasing down a long ball, was clumped with a high foot by an onrushing Moore. The goalkeeper was swiftly dismissed and the first thing his replacement, Jake Eastwood, had to do was pick Cavaleiro\u2019s deflected free-kick out of the net.\n\nThe scorer was then promptly substituted and replaced by Benik Afobe, making his second debut for the club after his loan move back from Bournemouth.\n\n\u201cNo doubt about it, we were beaten by the better side,\u201d Wilder said. \u201cThe quality of the first goal set a marker for the match. They\u2019re a fantastic side and they\u2019ve done that to other teams just as much as they\u2019ve done it to us.\n\n\u201cYou look at the quality on the field and the quality they have on the bench and it\u2019s not surprising. But for us, we were told this was going to happen to us every week when we came up. But this is 3 February and it\u2019s the first time it\u2019s happened.\u201d\n\nWith Derby winning 3-0 at home to Brentford, Wolves remain 11 points clear at the top of the Championship but in reality they are as far away from their competition as Manchester City are in the division above them. The quality of their football and plethora of attacking options suggest Nuno\u2019s team will not be frightened if \u2013 and almost certainly when \u2013 they make the step to the top flight.", 
    "created_at": 1517688607936, 
    "description": "Goals from R\u00faben Neves, Diogo Jota and Ivan Cavaleiro helped Wolves to a sparkling 3-0 win over fellow promotion hopefuls Sheffield United", 
    "document_end_position": 3272668, 
    "document_filename": "2018/2/4/7/1517727699_226714", 
    "document_start_position": 3105935, 
    "domain_id": "13012715934034554483", 
    "feed_id": "16602553348779256418", 
    "googleplus_source": null, 
    "googleplus_type": null, 
    "id": "17235469562661377896", 
    "image_url": "https://i.guim.co.uk/img/media/41adde6553aac61ca7560aa5c955aafa1b15d774/0_103_4166_2499/master/4166.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&crop=faces%2Centropy&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxOC8wMS8zMS9mYWNlYm9va19kZWZhdWx0LnBuZw&s=6c3b9d3a032088eab85d76c763281494", 
    "language": null, 
    "links": null, 
    "published_at": 1517687274000, 
    "social_fb_comments": null, 
    "social_fb_likes": null, 
    "social_fb_shares": 1100, 
    "social_gpluses": 0, 
    "social_linkedin": 0, 
    "social_pinterest": 0, 
    "social_scheduled_at": 1517965903078, 
    "social_total": 1100, 
    "social_tweets": null, 
    "social_updated_at": 1517865069044, 
    "social_updates": [
      {
        "linkedin_shares": 0, 
        "pinterest_pins": 0, 
        "google_plus_ones": 0, 
        "updated_at": 1517713827000, 
        "facebook_shares": 569, 
        "total": 569
      }, 
      {
        "linkedin_shares": 0, 
        "pinterest_pins": 0, 
        "google_plus_ones": 0, 
        "updated_at": 1517814652000, 
        "facebook_shares": 1000, 
        "total": 1000
      }, 
      {
        "linkedin_shares": 0, 
        "pinterest_pins": 0, 
        "google_plus_ones": 0, 
        "updated_at": 1517865069000, 
        "facebook_shares": 1100, 
        "total": 1100
      }
    ], 
    "tags": null, 
    "title": "R\u00faben Neves a class apart in Wolves\u2019 mauling of Sheffield United", 
    "twitter_source": null, 
    "twitter_type": null, 
    "types": [
      "article"
    ], 
    "updated_at": 1517728152190, 
    "url": "https://www.theguardian.com/football/2018/feb/03/wolverhampton-wanderers-sheffield-united-championship-match-report", 
    "word_count": 671
  }
]

Use case:

• Crawling and indexing posts, updating social metrics over time in social_* and social_updates.
• Tracking authorship.

[chaudum]

Hey @jeffnappi if there isn’t anything we can do now, please feel free to close the issue. It can be re-opened any time again.

[chaudum]

I checked the logs and I do not see any consistent issues with locked shards - there was only one org.elasticsearch.env.ShardLockObtainFailedException in the past 3 months and there were no other mentions of issues relating to locks that I could see.

That is at least good news. 😃

Regarding the REROUTE - are you proposing a workaround wherein I’d move the oversized shards to a different node which would presumably free up the extra space used on the source node?

Yes, the moved shard should be the “regular” size then, making some more free space for the other shards to merge segments.

One thing that came to my mind was increasing the indices.store.throttle.max_byte_per_sec setting to a something more SSD friendlier value: 200mb (default is 20mb). This setting controls how much bytes of segments are merged in the background (because it requires disk i/o and cpu). If this is too low and you create a lot of new segments (when you do a lot inserts/updates/deletes) merging might fall behind.