A more advanced form of coding query involves the “Coding Comparison” feature.  This query enables a researcher to compare the similarities and differences between how individuals or teams code in a shared project. 

In general, if a research team wants to see if a particular social construct is supported as something extant (instead of reified), a team would train together on how to code around a shared codebook. They would strive towards controlling against rating variability.  They would engage a shared project and would would have a common codebook and shared data for coding...or at least something in common, like a codebook OR shared data...or some other combination.  Then, if they independently code the data with a statistical level of similarity, they can perhaps make the argument that the construct appears to exist... 

For some projects, especially if a team’s members are all coding different parts of a shared dataset and need to be trained and normed to code similarly, then a high Kappa Coefficient (a sort of inter-rater reliability based on the similarity analysis of their respective coding) is desirable.  If a team is collaboratively coding the same data (with all members with the same data set), then it may be more valuable to focus on dissensus and creative interpretations—so that new ideas may be surfaced.  Research questions and objectives, theories, methodologies, and domain-based approaches will often vary widely.  The unofficial e-book focuses on the tool but not really how the findings may be interpreted. 

This tool enables dyadic comparisons only, but multiple coders may be placed on Team A or Team B for melded comparisons.  Or one coder may be used as a baseline, and the other coders may be compared against that baseline coder. 

Another source offers a table that summarizes how to interpret the Kappa coefficient.  (Do note that different topical areas set the relevance threshold for "similarity" at different levels. 

Multilingual data will not be handled as easily in terms of comparisons, or they may create their own anomalies.  [This is especially if there is use of base language coding by NVivo for autocoding for topic modeling (themes and subthemes) or sentiment (positive - negative sentiment).  In such cases, one "text content language" is used for the extraction, so if a project has multiple languages, the Kappa will be lower...since the computer is only using one main language for its autocoding.]  

To offer a more accurate sense of a coding comparison query, the following is a screenshot of the example that QSR International has made available on its site. 

This Coding Comparison results in a table with both nodes and sources (depending on how the researcher set up the parameters of this Coding Comparison).  In other words, it is possible to look at Kappa coefficients at each node and each source item level.  Nodes, if there are no overlapping codes (e.g. A coded some text to the node, but B did not), will be at 0 (because there is no similarity between the coding behaviors of the two sides for that particular node).  

Also, please note that this procedure may result in exportable data as a table.  The exported table data may be analyzed in other tools, such as those that may enable more in-depth statistical analysis.  .  

A Summary Kappa's Coefficient:  Also, the Kappa Coefficient is compared across coding nodes and individual sources. For a summary Kappa Coefficient, export the list (right-click the table and click "Export List"), and run the summary Kappa in Excel (by running an average on Column G...).  

Researchers can sometimes have problems when trying to run a coding comparison across projects from different coders.  Problems may be introduced when project ingestion is done incorrectly.  

For example, if a master project was not created with the desired codes and labeled sources, then it is possible that each of the coders used different node naming protocols and different source naming protocols.  To salvage the work, the person or team integrating the stand-alone projects will need to merge the nodes to one correct set...and delete the incorrectly named nodes.  Likewise, if there were problems in the naming of sources, then that may be even more difficult because then the actual sources have to be corrected.  (If this is the case, I would return back to the original coded project and re-name the sources for full alignment with the master project...before importing the project...into a master file.)  

Once this norming has been correctly done, it is then possible to run a correct Coding Comparison Query...with an accurate Cohen's Kappa.  

A basic similarity matrix structure follows:  

Coder 1/ Coder 2	Attribute 1	Attribute 2	Attribute 3	Attribute 4	Attribute 5	Attribute 6
Attribute 1
Attribute 2
Attribute 3
Attribute 4
Attribute 5
Attribute 6
Total
Percentage Agreement

There are some important caveats here.  The Kappa number (generally conceptualized on a scale of -1 to +1) is affected by the underlying dataset.  If there are a large number of coding nodes, and the data is highly noisy and complex, the Kappa could be much lower than in a project in which the coding nodes are fewer and a dataset which is simpler. This is the reason why many researchers argue that you cannot compare Cohen's Kappa across datasets unless  they are somehow "similar".  With qualitative, mixed methods, and multi-methodology types of datasets, comparability of datasets is an even more difficult problem.  

Also, if team coding is applied in an exploratory way by research team members with individually very different objectives and skillsets and backgrounds, low Kappa would likely be desirable to capture coding divergence, dissensus, and non-agreement.  There are times when it is beneficial to have a broad range of understandings when approaching a project. For example, if a team is building a consensus codebook in a bottom-up coding way using a wide range of contents, divergent coding (and initial dissensus) may enable the collection of a wide range of concepts for the codebook.  (The assumption here is that the node structure set up in the NVivo project used is broad and general to be inclusive of the wide range of interpretations.)  Having a myopic and limited codebook would be a limitation to the project. 

Finally, the coding comparison here evokes something of multi-coder co-orientation towards a particular topic.  There is a co-orientation research model from the 1970s. 

Also, note how visuals make data relationships feel intuitive.  (Lots of practice in using and viewing these data visualizations may improve analytical depth.)