Chess computing resources for 2015 - Part I: Database and Analysis Software

This starts a series of posts that will be, in part, an update of previous ones on computer-related resources.  In this post, I intend to review chess database and analysis software functionality in a more in-depth manner, from the point of view of an improving club player.  My primary aim is to provide a reasonably comprehensive guide on how improving players can use these computer tools in a practical manner.  I therefore focus on how they can assist game analysis and related studies, which I've made the centerpiece of my own improvement program.

Online chess game publishing is a separate but related topic that I think deserves its own future update, although the last post on the topic from 2013 remains useful as a starting point.

Companion posts in this series on other types of computing resources can be found here:
Part II - Chessplaying software and computer opponents
Part III - Game databases and e-books
ChessBase's list of web-based applications

Database and Analysis Software

This category of computing tools, which use game databases for training, study, research and analysis purposes, has evolved to the point where I believe it can and should be employed by all serious chessplayers, not just the pros.  The broad range of functions now included in software packages cover just about everything you would want to do in chess analysis and preparation.

Although some database programs offer options for computer opponents along with analysis functions, that is normally not their primary focus, so I've decided to break that category out (along with a more detailed discussion of computer engines) into a separate post on chessplaying software / computer opponents.  Finally, in a third post I'll include a list of some game database resources and discuss the use of chess e-books (that in reality are annotated databases) and related software.  Naturally, individual software packages can contain all of the above categories of chess resources.

Below we will explore various types of database functions and highlight their possible utility for the improving player.  In large part, I'm doing this in order to help expand my own training tools for the new year by making myself smarter on the available options.  That said, I hope readers may also find the walkthrough and function review helpful.

Because ChessBase 13 (CB13) was just recently released and is popularly viewed as the "gold standard" for database software, I'm using that as the primary reference point for examples.  Other packages (including the two listed below) may provide similar functionality in many respects and for less money (or even for free, in the case of SCID vs PC).  Because of the quality freeware available, there is now no barrier to entry for any serious player (of whatever grade) to have, at minimum, a database of their own collected games available for analysis, along with a large database of master-level games for study, comparison and research.

• Chess Assistant 15 - this package from ChessOK is the main commercial database alternative to CB13, with its own proprietary format, although both it and CB13 can also work with PGN files.
• SCID vs PC - this is a full-featured freeware database analysis package (including the Stockfish engine and pre-loaded opening books) that also has some extra features like a FICS client for online play.  Latest release version as of this post is October 2014, from the Sourceforge site linked above.

For organizational purposes, below I'll examine different database functions under the headings of opening, middlegame and endgame study, along with a final section on game analysis, which serves to put it all together.

A.  Opening Study

One of the fundamental tasks for a chessplayer is to keep a record of their opening repertoire, as it is created and evolved.  Although ChessBase has offered designated "repertoire" databases with the most recent versions, including two individual ones for White and Black with CB13, this type of database organization and use is not terribly intuitive, including for more advanced players.  The simple openings database system I use - SORDS - organizes different major variations into games, similar to the way openings books are divided into different chapters. This is something any database program can do for you.

• Collect Openings is a CB13 function (see PDF manual p. 227) that is designed to be used on a database of your own games.  It will automatically create a new database that groups and merges games into different opening classifications, in effect showing you what your repertoire has been.  This seems like a powerful tool and could, for example, be used to generate your first repertoire database automatically.
• Opening "books" are included in many database programs, can be generated from them, or downloaded as separate files.  CB13 allows you to generate them from a reference database (see PDF manual p. 191) or if you are online to access the "Live Book" which is always up-to-date.  A "book" gives you a tree of opening moves to follow, similar to printed references such as Modern Chess Openings (MCO), the Encyclopedia of Chess Openings (ECO, whose codes are now standard), Nunn's Chess Openings (NCO), etc.

Statistics are normally included along with book displays. For example, here is a screenshot of the LiveBook from CB13.  It shows the move (the opening move for White, in this case), number of games in the book with that move, the score (result) in percentage terms, average Elo of players playing that move, the date, and the book evaluation.  This is typical across database programs.  The CB13 Livebook also has "visits" and a percentage next to it, which shows what moves people have been looking at in the online book.

The "Reference" tab in CB13 provides a deeper look at the position on the board and is primarily used for study of the opening phase, as it automatically retrieves and compares all games in the database with the same position.  As an example, below shows Gelfand-Tomashevsky from the 2014 Baku Grand Prix commentary after Black's 6th move.  As with the book format, we can see the move list (starting with White's move 7), number of games in the database and score (from White's perspective).  It also includes the last time it was played in the database by a titled player, the "Hot" indicator (a bar that reflects more recent play by top players), and a list of the best players who have played the move.  (move list, % result from White's perspective, "Hot" games, best players)

In the second window, the database program provides the most popular and highest-scoring lines (showing number of games and percentage score) along with the line fragment.  This shows at a glance how the game can evolve from the position and points of departure.  The third window contains all of the database games (obviously with a scroll bar) and can be sorted by all of the column headers - White player, White Elo rating, Black player, Black Elo rating, result, date, and notation.  These functions are all quite helpful and will be discussed further in the Game Analysis section below.  All of the games shown can be brought up in a separate window by double-clicking them.

"Report - Opening Report" is a CB13 function that scans the reference database and sorts information into a new database with HTML text and links, essentially providing an automatically-generated document detailing the reference pane information.  While this can be quite helpful to have all laid out for you in once place, as the main moves and continuations are included in the output, along with links to high-level sample games, the function is not fully reliable.  For example, running the opening report on the game position above, from October 2014, does not include it or any of the other 2014 games in the database output.  I believe this is a quirk in how CB13 is querying the date field of the games, as I've noticed the 2014 games in the database (Mega 2014) have included a month in that field; the software appears not to be able to recognize that with this particular function.  This same quirk can affect column sorting by date, although the games themselves at least appear.

Search functions allow you to look up games by ECO code and whatever other criteria you would like to add.  For example, as part of my "Stonewall Hero" exercise, I pulled all of Artur Yusupov's games in the Stonewall (ECO code A90) up via the search function and copied them into a new database.

• In the reverse process, most databases should allow you to look up the ECO code of the game - in CB13 this can be done via "Report - Opening Classification".  ECO codes can also be searched online at various chess sites (or even just via standard internet search such as Google, for example "ECO code Classical Caro-Kann").

Identifying the game novelty (the move when it first goes out of the database) can be done various ways, including simply stepping through the game moves until there are no other database games showing in the Reference section.  The shortcut for this in CB13 is to use the Report - Novelty annotation function, which accesses the Live database for comparison purposes.  In the example above, Gelfand-Tomashevsky, the novelty is flagged as 12...c5 and the software automatically includes a predecessor game as a variation in the annotations, as well as a game in what it considered a relevant major variation for White on move 9.  More on the role of variations will be discussed in the below Game Analysis section.


B.  Middlegame Study

While openings are an obvious choice for database research, since there are normally so many other examples to draw on for each game analyzed, the middlegame should not be ignored either.  Looking at similar structures, tactics and maneuvers across different games is an excellent way to identify and better understand position-types that are common in your games.

In CB13 the user has to manually activate the "themes" for database sorting, but once that is done (File - Options - Misc - Activate theme keys), they will automatically appear for databases in ChessBase format.  There are four tabs with different theme categories: a general one, tactics, strategy and endgames.  What appears there will depend on how (or if) the games have been modified or marked.

• "Themes" tab contains a number of different miscellaneous categories, the most useful of which (if you have an annotated database like Mega 2014) is probably "Commented games", which contains games with comment symbols and/or text broken down into a number of subcategories such as "Initiative", "Attack", etc.  Individual historically important players have their own subcategories, for example in Mega 2014 Fischer has 162 commented games.
• "Tactics" tab provides broad categories ("Attack", "Defense", etc.) rather than examples of specific tactics - see below for a more specific search function - but also includes games with training questions.  In Mega 2014 there are 3732 games with training questions (2088 commented) included.  Because CB13 allows you to edit any game to include training questions, this could be a very useful tool for assembling your own computer-based "flashcards" for tactics, using your own games or those of others as a base.  Training questions can also be used in an opening repertoire database, as a means of testing your recall and understanding of positions.  The questions can also be toggled on or off in the main menu, so if you prefer to see the whole game at once it's not an obstacle to have the questions included in database games.

• "Maneuver search" (PDF manual p. 88) is the way you can look for a particular tactic across different games using CB13.  You need to follow the manual's instructions for the search, but it allows you to do some powerful things, for example look for all bishop sacrifices on h7 while automatically mirroring tactics for the other side (in this case Black bishop sacrifices on h2).
• "Strategy" tab contains a large number of categories and is much more developed than the corresponding tactical themes.  For example, there is a detailed breakdown of pawn structure types and different material imbalances.  This offers an easy way of researching both classic and recent games with particular middlegame features, for example an isolated White pawn on d4.

•  "Report - similar move" (PDF manual p.168) - "With one click the program finds games with similar pawn structures where similar moves were made."  When analyzing games (see below), this is a particularly interesting feature.
• "Report - similar structures" (PDF manual p.167) - "With one click the program loads all the games with similar pawn structures from the database and sorts them by similarity to the current game. The program also takes account of the positioning of the pieces (rooks on open rows, queen and bishop on the same diagonal, the position of the kings)."  This is also a particularly powerful comparative feature when analyzing games and can benefit a serious student, one who wants to emulate Benedict of Amber and observe how similar battles are fought.  Both this and "similar move" do appear to work properly, in my limited testing.

C.  Endgame Study

Endgame themes lend themselves well to database analysis and comparison.  The "Endgames" tab contains a lot of different subcategories, for the most part broken down by material, already populated.  Below I've expanded the Pawn Endings and Rook vs. Other Pieces endings as examples.

• "Report - Similar endgames" is a CB13 function that can be called from a specific game (PDF manual p.167) - "The games are sorted by similarity. The program takes account of pawn structures (passed pawns, blocked pawns, connected pawns, chains, isolanis, backward pawns) and the relevant positioning of the pieces (rooks behind passed pawns, rook is cutting the king off, king in a square, wrong bishop, etc)."  Again, this type of function is useful for your own game analysis, especially if you want to take it to a deeper level or to get a quick idea of how other players have handled similar situations.  This can, among other things, help identify winning ideas and plans.
• Searching for material distribution in an endgame is a more basic function, but if your database software lacks automatic report generation, then it can serve a similar purpose.

D.  Game Analysis: putting it all together

For the type of game analysis I normally do for training purposes, I take a balanced approach between doing only a superficial review and a deep probing of my games and selected master-level ones of interest.  For a more detailed discussion of the benefits of the analysis process, you can follow the above link; here I'll focus on the utility of using database and analysis software to enable the process.

• Commentary / "your thoughts here" - any software that allows you to create and maintain a database of your own commented games is an invaluable study tool.  Putting your thoughts down (in terms of text commentary, symbols, and variations) shortly after each serious game you play is fundamental to better understanding your strengths and your areas needing improvement.  Normally I do this on a fairly superficial level (taking 15-20 minutes) as a first pass on analysis of one of my games, to identify my thinking process, evaluations during the game, and key variations that were considered but not played.
• Openings / "what would Grandmasters do?" - the next step in analysis is stepping through the opening moves with the "Reference" tab open.  This tells me how far the game follows the main line (as indicated by the majority of database games at the master level), if there are major move branches or only one overwhelming preference by professional players, and helps capture transpositions that only looking at the move sequence would ignore.  Some of the most useful additional functions are looking at what the strongest players for your side do (sorting by Elo) and also sorting by date, to see what is currently considered best (or at least most popular).  Other games can easily be copied into your game as variations, either for temporary or permanent reference.
• Variations / "choices choices choices" - using the software, you have a free hand to put down all of the options that occur to you at each point and then explore them as much as you want.  While the same type of thing can be done using a physical board - and this has its uses for training - the virtual board gives you much greater efficiency in moving between the stem position and all of the various branches, as well as recording your thoughts.  It's important to be curious about what works and what doesn't in a position (and why).  Something that made an impression on me was a quote from GM Nigel Davies, as blogger Robert Pearson used to cite, "...move the pieces around...Grandmasters do this, amateurs don't."  In other words, set things up for yourself and work it out.
• Engine analysis / "evaluate" - there has been a good deal of silliness written about the horrible evils of using chess engines in training, on the one hand, and their infallibility, on the other hand.  "Pitfalls of Computer Analysis" contains some more detailed thoughts on this topic.  To my mind, the best use of engines, when loaded into database software, is to provide a very strong objective evaluation of the position in front of you.  One trick even top grandmasters like Kramnik use is to load an engine and have it display its current evaluation, but not the moves under consideration, so they can work it out for themselves.  The major caveat for improving players using an engine for assistance is that, if you cannot understand its evaluation in the absence of concrete tactics, it will not be of much use to you from a learning perspective.  However, if you can at least describe the major positional factors at play, a top engine can help guide you in better understanding things like the value of compensation, space advantages and structural strengths/weaknesses, especially as you step through variations to see their final outcomes.  Engine analysis normally is of significant assistance in identifying key turning points in a game, along with highlighting missed opportunities - especially ones which would simply never occur to you.

For more in-depth computer analysis, CB13 includes the "deep analysis" function, which is best left running for a lengthy period of time on a game and then saved as a different game from the original one.  It also goes a step further with programmed multiple "analysis jobs" which let you queue up position analyses with various advanced parameters for your engine(s).  This type of approach was first included a few years ago in the Aquarium software package (which will be referenced in the next post in the series) as "IDeA" projects.  These types of functions in practical terms will be of interest more for the professional or correspondence player, or people who are more interested in very deep analysis of individual games; I have not done more than give the functions a cursory look.

In Part II of this series, I will take a closer look at chessplaying software and engines, including some additional thoughts on the practical value of computer opponents and analysis for the improving player.

Below are some additional references for CB13 functions:

• ChessBase 13 through the eyes of a top Junior - I found this to be the most useful of the articles so far in understanding the utility of some of the functions.
• ChessBase 13 from a club trainer's perspective (1) - this is another good walkthrough of CB 13 functions, with lots of screenshots.
• ChessBase 13 from a club trainer's perspective (2) - part 2 of the walkthrough, which has additional screenshots and function discussions beyond the scope of this post.
• ChessBase 13 from a club player's perspective - a more basic walkthrough of features.

All of the above are from the ChessBase site and of course one must keep in mind that they are designed to help sell the product.  The individual authors' assessments of the product nevertheless seem to be valid; I would consider that the first article is the most candid in that respect.