Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats

SongHo Sim; SinHyok Kim

doi:doi:10.11648/j.ajetm.20251005.11

Research Article |

| Peer-Reviewed

Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats

SongHo Sim^*

, SinHyok Kim

Published in American Journal of Engineering and Technology Management (Volume 10, Issue 5)

Received: 14 September 2025 Accepted: 30 September 2025 Published: 31 October 2025

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

The group decision-making (GDM) is a process of aggregating the preference information of each decision maker in a group into a collective opinion under certain decision criteria. When a decision is made by decision makers (DMs), the group decision or group opinion is a result of the integration of the individual opinions by a mathematical aggregation. An important thing in the integration of the individual opinions is which DMs’ opinions should be of importance in the aggregation process. The group opinion is heavily influenced by the level of expertise of the DMs. In real-life group decision making (GDM) problems, preference relations given by decision makers (DMs) are often heterogeneous because of their expertise and different decision habits. The expertise level of the DMs can be estimated using the ‘the ability to differentiate consistently’, a ratio between discrimination and inconsistency. Each expert has different expertise in different criteria and has his/her limited capacity in constructing of pairwise comparison preference relations. In this paper, we focus on a method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats. The combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations is discussed, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs, including FPRs under certain criterion. Then, since the experts have the limited capacity in constructing of pairwise comparison preference relations in GDM with multiple criteria, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of judgements. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Numerical analyses show that proposed method is simple than the previous methods and can fill up some gaps in GDM.

Published in	American Journal of Engineering and Technology Management (Volume 10, Issue 5)
DOI	10.11648/j.ajetm.20251005.11
Page(s)	69-83
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2025. Published by Science Publishing Group

Keywords

Group Decision Making, Ranking, Heterogeneous Preference Relations, Expertise

1. Introduction

Weiss and Shanteau

[1]

proposed the concept of ‘the ability to differentiate consistently’, which is the ratio between discrimination and inconsistency, to define the DMs’ level of expertise. They defined experts as those who are capable of distinguishing between cases that are similar but not exactly the same and of repeating their judgments consistently, and proposed the CWS-Index (the Cochran-Weiss-Shanteau Index), which is the ratio between discrimination and inconsistency, to assess expert’ s level of expertise

[1, 2]

. The CWS-Indexes for the experts yields their ranking; the higher the CWS-Index, the higher is the DM’s ranking. However, measuring inconsistency requires repetition, and accordingly the experts need to make judgments more than once.

[4, 5]

, the researchers pointed out the irrationality of the repeated evaluation to measure the inconsistency of experts judgements, the authers in

[6-11]

showed that the additive consistency (AC) of fuzzy preference relations (FPR) could be used to measure the expert’s consistency level. But these studies don’t consider the ability of the expert to differentiate between similar, but not identical, cases, and produce a experts’ ranking based on consistency of the FPRs.

There have been studies to determine the ranking of experts based on their level of assessment, but in these previous researches, the experts’ ranking are determined only by the consistency of their assessments, without considering their ability to differentiate.

In sum, the previous studies have not used the comprehensive concept of expertise.

[3]

, Herowati et al. proposed a method to determine the raking of the DMs using the combination of expertise as ‘the ability to differentiate consistently’ and the AC of FPR in GDM with one criterion.

In real-life GDM problems one be often faced with following some situations.

1) The experts may have different expertise for different criteria

[3]

2) The preference relations provided by experts are often heterogeneous because they always have different decision habits

[12]

3) The experts have their limited capacity in constructing of pairwise comparison preference relations.

To assess expertise-based ranking of experts for these situations appears to bear a certain meaning in the study of GDM problems. Furthermore, if expertise-based ranking of experts is developed into expertise-based experts’ importance weights that specify the importance weights of the experts in GDM, we can use these importance weights in the process of aggregating the individual opinions into a group opinion. Research to obtain the experts’ importance weight from their ranking has been discussed in another paper

[13]

In this paper, we focus on a method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats.

The combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations is discussed on the basis of the methodology proposed in

[3]

, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs, including FPRs under certain criterion. Then, since the experts have the limited capacity in constructing of pairwise comparison preference relations in GDM with multiple criteria, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of judgements. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Numerical analyses show that proposed method is simple than the previous methods and can fill up some gaps in GDM.

The remainder of this paper is organized as follows. In Section 2 we introduce related preliminaries about expertise levels of experts and different preference relations. In Section 3 we refine the method for ranking the expertise levels of experts when experts provide their judgements for alternatives in MPRs, utility values, preference orderings and incomplete preference relations under one criterion. In Section 4, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in multi-criteria decision making (MCDM). In Section 5 we apply expertise-based ranking of experts to aggregate individual opinions into a group opinion in GDM. In Section 6 a numerical example is discussed to illustrate proposed method, and we conclude the paper in Section 7.

2. Preliminary

In this section, we firstly discuss the previous methods related to the expertise level of the experts, then introduce different preference representation formats often used in GDM.

2.1. Experts’ Expertise Level and FPR’s AC

Weiss and Shanteau

[1]

proposed to combine the concepts of ‘consistency’ and ‘discrimination’ to determine the expertise level of a person that becomes ‘the ability to differentiate consistently’, which is expressed by the CWS-Index as shown in equations (1), (2) and (3) as follows:

(1)

(2)

(3)

where

: The number of replications,

: The average of individual values for case-

: The grand mean of all individual values,

: The number of different cases,

: The individual value for replication-j of case-

Equation (2) shows that discrimination expresses the compatibility variance in statistics, and equation (3) shows that inconsistency is the reproducibility variance.

In equation (1), to get the CWS‐Index, experts are asked to give their assessment repeatedly.

Repeating the measurements are difficult and time-consuming. It’s another weakness lies in the possibility to obtain a high CWS-Index score by giving an incorrect assessment consistently. Therefore, the method for comparing experts’ expertise needs to be refined.

FPR is one of the most widely used evaluation methods for expert assessment in GDM

[8, 14-18]

We assume that

is a set of feasible alternatives to be assessed.

A FPR is determined by a pairwise comparison matrix (PCM) whose entries represent preference degree for two alternatives. The values of a fuzzy PCM

range from zero to one and satisfy

. If

is 0.5, then the alternatives are equally important. If the value of

(resp.

,) is one (resp., zero), then the alternative

is the most (resp., least) important than

The AC for FPR can be expressed with the following three relational equations, therefore each entry of the fuzzy PCM

is estimated in three different ways.

(4)

(5)

(6)

Since FPR has property

, results estimated from equation (4), (5) and (6) are the same, and for every entry of the FPR

, the equations produce as many as

estimators (since

All entries

of the estimated matrix can be transformed using a transformation function such that the range changes from

[8]

(7)

[3]

, Herowati et al. refined the method to measure a person’s level of consistency and compare the expertise level of the experts on the basis of the deviation between the values of the estimations using the AC of FPR and the real values given by the expert. In terms of combining the concept of expertise as ‘the ability to differentiate consistently’ with the AC of FPR, CWS-Index is revised as follows:

(8)

(9)

(10)

where

: The number of replications composed of one real value and

estimators

: The average of individual values for case-

: The grand mean of all individual values

: The number of different cases

: The individual value for replication-

of case-

2.2. Other Preference Representation Formats

(1) Utility Values

Assume

is the set of utility values provided by one of the experts.

represents the utility value corresponding to alternative

. Generally,

range from 0 to 1, and the higher the utility value, the more important the alternative possesses

[14, 19-21]

(2) Preference Orderings

Let

be a preference ordering set. This set is the permutation function over the set

. For example, if an expert provides the ordering

for four alternatives

, then the preference ordering is priority sequence of alternatives, that means the alternative

is the best selection among the candidate alternatives, and

is the least preferable of the alternatives

[14, 19, 20, 22]

(3) Multiplicative Preference Relation

[22-24]

In contrast to the utility values and preference orderings, the multiplicative preference relation is always represented by a PCM. Assume the matrix

is the PCM provided by experts. The entry

of the PCM is the degree of preference for alternative

over

. The multiplicative PCM of Saaty

[24]

satisfies

and

The consistency for the MPR of Saaty

[24]

can be expressed as follows:

(11)

3. Ranking of Experts’ Expertise According to Different Preference Relations

In this Section we refine the method for ranking the expertise levels of experts when experts provide their judgements for alternatives in MPR, utility values, preference orderings and incomplete preference relations under one criterion.

3.1. The Case of MPR

Suppose MPR

is given. MPR

can be converted into FPR

under the transformation

. Then we can rank the experts’ expertise level using the method in

[3]

. However, we present the method to directly rank the experts’ expertise level from MPR

, referring to the method in

[3]

According to equation (11), the consistency for MPR can be expressed with the following three relational equations, therefore each entry of the MPR

is estimated in three different ways.

(12)

(13)

(14)

Since MPR has property

, the same results are estimated from equation (12), (13) and (14), and for every entry of the MPR

, the equations produce as many as

estimators (since

). A expert’s level of consistency can be used to measure based on the deviation between the values of the estimations and the real values given by the expert. The consistency level is then used to generate consistency-based ranking of experts.

We perform the following transformation procedure to keep the range of each element

of the MPR matrix

within the interval

1) Perform the logarithm transformation to the base 9 for

and the corresponding

values of estimations

2) Perform the following transformation so that we have

3) Perform the antilogarithm transformation for

to yield

Now we can calculate the CWS-Index from equations (8), (9), (10) to rank the expertise levels of experts.

3.2. The Cases of Utility Values and Preference Orderings

Suppose

, the set of utility values is given. We can form MPR

in terms of the pairwise comparisons between the utility values,

. Furthermore, MPR

can be converted into FPR

under the following transformation.

Suppose

is the preference ordering set. For preference ordering, in

[25]

Herrera et al. proposed a nondecreasing function

that assigns ordering values to utilities as follows:

Then MPR

can be formed in terms of the pairwise comparisons between the utility values,

as follows:

(15)

For four alternatives

, if an expert provides the preference ordering

, then

have the indeterminate forms from equation (15). Even if

is replaced by 0.0001, when

, we can not distinguish the case of

. Hence, we revise the equation (15) as follows:

(16)

Therefore, in the cases of the utility values and preference orderings, the expertise level of experts can be assessed using the equations (8), (9), (10).

3.3. The Case of Incomplete MPR

If at least one element in MPR

provided by individual expert is unknown, then

is called the incomplete MPR

[26]

. It is prerequisite for the incomplete MPR to estimate missing pairwise preference values. The values of the estimations obtained from equations (12), (13), (14) can be used to complete the incomplete MPR matrix. In the paper we propose a method to estimate the missing values in the incomplete MPR by modifying a little the equations (12), (13), (14), and study necessary properties.

Each element of MPR

from equations (12), (13), (14) can be estimated as follows:

(17)

(18)

(19)

Obiously, when the elements of MPR

are the crisp values, the same results are obataind from equations (17), (18), (19). But it is not so in the case of the interval MPR. So, in order to study hard to the case of the interval MPR in the future, we use single estimated equation by integrating equations (17), (18), (19) as follows:

(20)

Theorem 1. Let

be a multiplicative preference relation. The matrix

is complete consistent if and only if we have

for

estimated from the equation (20).

Proof. It is obvious for the necessity. One can rewrite the equation (20) as follows:

(21)

We have

from the equations (17), (18), (19) and

. Therefore

hold from the equation (21) and we can see

from the condition

. That is,

This is written in form of matrix as

A^{T} a = na

, where the vector

a

denote

a = (a_{11}, a_{12}, \dots, a_{1 n})

. In consequence,

(A^{T} - nI) a = 0

, where

denote the

-identity matrix.

Since

a

is not the zero vector, thus

has the largest eigenvalue

As proved already, that matrix

is complete consistent if, and only if, we have

. As a result,

is complete consistent matrix.

Now we build an algorithm to estimate the missing elements of the incomplete MPR

. First, we will introduce the following notation.

: set of all pairs between alternatives

and

: set of the missing elements of

: set of the known elements of

The values of

can be estimate using the following equation, where

|∙|

denote the cardinality of the set.

(22)

Next, the following algorithm is developed to estimate the missing elements of the incomplete MPR

Algorithm 1.

Step 1: Let

and

Step 2: Fist, find

. Then, obtain the set of pairs of indices

that can be get newly in iteration

using the equation (22), where

denote the set of the missing elements of the matrix

after iteration

Step 3: If

, then terminate algorithm, otherwise use the equation (22) to

and put the estimated value

to the matrix

. The value of

is given by

, and then set

; go to step 2.

Remark 1. In step 3 of algorithm 1, the case of

has two states. One is the state in which we obtained all the missing elements of the matrix

, and another is the state in which one or more unknown alternatives come into existence. Except the unknown alternatives, the values of pairwise comparisons between others can be estimated through algorithm 1.

First, let’s consider the state that there exist one unknown alternative. Assume that first alternative is unknown without loss of generality. If we suppose that

, then from the equation (22) the elements in first row can be expressed as follows:

As seen above, the coefficient

in recurrence equation

can be find, since it is composed of the values of pairwise comparisons that are already known. If we assume that

, then

must be hold for any

. That is,

must be hold. Thus one specifies arbitrary

such that

. The elements in first column are calculated from

Theorem 2. Let all elements in one row of the incomplete MPR

be unknown. Then the largest eigenvalues of the estimation matrices corresponding to arbitrary

are the same.

Proof. Suppose all elements in first row of the incomplete MPR

are unknown, then we estimate the missing elements among the rests except first row and first column by algorithm 1. Let

be the estimated matrices corresponding to arbitrary

, and let

be those largest eigenvalues, repectively.

If we assume that

, then we have the following relations between the matrices

and

Let

w = {(w_{1}, w_{2}, \dots, w_{n})}^{T}

w^{'} = {(w_{1}^{'}, w_{2}^{'}, \dots, w_{n}^{'})}^{T}

be the vectors of the weights obtained from the matrices

and

by applying the row geometric mean prioritization method

[24]

. Since

{λ (A) \frac{1}{n} \sum_{i = 1}^{n} \frac{{(Aw)}_{i}}{w_{i}}}_{\max}

, if we show that

\frac{{(Aw)}_{i}}{w_{i}} = \frac{{(A w^{'})}_{i}}{w_{i}^{'}}

is hold for all

i \in \{1, \dots, n\}

, then the proof of theorem is completed.

Let

be the row geometric mean of the matrices

, and let

be its’ sums, respectively. Then its’ weights are determined by

. From the relations between the matrices

, we have the following equations.

Then, for arbitrary

we have

\frac{{(A w^{'})}_{i}}{w_{i}^{'}} = 1 + \sum_{j = 1, j \neq i}^{n} a_{ij} \cdot \frac{w_{j}^{'}}{w_{i}^{'}} = 1 + \sum_{j = 1, j \neq i}^{n} a_{ij} \cdot \frac{\frac{{w^{'}}_{j}^{*}}{S^{'}}}{\frac{{w^{'}}_{i}^{*}}{S^{'}}} = 1 + \sum_{j = 1, j \neq i}^{n} a_{ij} \cdot \frac{{w^{'}}_{j}^{*}}{{w^{'}}_{i}^{*}} = 1 + \sum_{j = 1, j \neq i}^{n} a_{ij} \cdot \frac{w_{j}^{*}}{w_{i}^{*}} = 1 + \sum_{j = 1, j \neq i}^{n} a_{ij} \cdot \frac{\frac{w_{j}^{*}}{S}}{\frac{w_{i}^{*}}{S}} = \frac{{(A w)}_{i}}{w_{i}}

Therefore, we have

. This completes the proof of Theorem 2.

From Theorem 2, we can know that the consistency levels of the estimated matrices corresponding to arbitrary

Similarly, one can consider even the case that there exist the unknown alternatives more than two.

In sum, the incomplete MPR

has become the complete MPR

. As a result, one can assess the expertise levels of experts by using the equations (8), (9), (10).

3.4. The Case of Incomplete FPR

If at least one element in FPR

provided by individual expert is unknown, then

is called the incomplete FPR

[27]

. In general, FPR can be converted into MPR under the transformation

. Therefore, given the incomplete FPR

, the corresponding incomplete MPR

can be obtain. Then, by the method in subsection 2.3 one can obtain the complete MPR

from the incomplete MPR

. As a result, from the the equations (8), (9), (10) the expertise levels of experts are assessed from

4. The Adjacent Pairwise Comparison Technique and Expertise-Based Ranking of Experts in MCDM

Let

be the sets of feasible alternatives and predefined criteria, respectively. When the experts provide their judgements for alternatives in FPRs and MPRs under

criteria, the above proposed method to rank their expertise levels requires

pairwise comparisons for each expert. As the number of alternatives increases, this method calls for increasingly more judgments. It is very difficult for the experts to do so many judgments. From such reason it need to propose the method to assess the expertise levels of experts on the basis of a small number of the judgements as possible. In this section, we propose the method to assess expertise-based ranking of experts using the adjacent pairwise comparison technique in MCDM.

Instead of the conventional technique which reqires pairwise comparisons for the given alternatives under each criterion, the adjacent pairwise comparison technique reqires only

pairwise comparisons between the mutual adjacent alternatives. When the experts provide their pairwise comparisons for the mutual adjacent alternatives in fuzzy and multiplicative preference format under

criteria, it aims to develop the method to assess expertise-based ranking of experts.

4.1. The Adjacent Pairwise Comparison Technique with Fuzzy Preference Format and Expertise-Based Ranking of Experts

The adjacent pairwise comparison technique requires for the experts to judge the relative importance between the alternatives corresponding to arbitrary

and

Let

be given set of experts. The adjacent pairwise comparison matrix which the expert

provides using fuzzy preference format is shown formally below.

where

are the judgements provided by the expert

. We estimate other elements in matrix

as follows:

(23)

Then, it can be showed easily that the matrix estimated from equation (23) satisfies the AC of FPR

[8]

Given the adjacent pairwise comparison matrices of all experts in fuzzy preference format, the adjacent pairwise comparison matrix of group can be build as follows:

where

, and other elements in matrix

are estimated as follows:

It is obvious that the adjacent pairwise comparison matrix of the group satisfies the AC of FPR.

Hence, when the experts provide their judgements in fuzzy preference format with the adjacent pairwise comparison technique, we can reform the CWS-Index for evaluating the expertise level of the experts as follows:

(24)

(25)

(26)

where

4.2. The Adjacent Pairwise Comparison Technique with Multiplicative Preference Format

When the expert

provides the adjacent pairwise comparison matrix in fuzzy preference format, it can be expressed formally as follows:

where

are the judgements that the expert

provides by vertue of a

-scale, and takes the integer from one to nine. Instead of a nine-point scale developed by Saaty

[24]

, the reason introducing a

-scale is that it does not satisfy an equal ratio relation between the mutual adjacent judgements. We estimate other elements in matrix

as follows:

(27)

It is trivial that the matrix estimated from equation (27) satisfies complete consistency of MPR.

Given the adjacent pairwise comparison matrices of all experts in multiplicative preference format, the adjacent pairwise comparison matrix of experts’ group can be build as follows:

where

, and other elements in matrix

are estimated as follows:

The adjacent pairwise comparison matrix of the group,

is also the complete consistent MPR. Therefore, when the experts provide their judgements in multiplicative preference format with the adjacent pairwise comparison technique, we can reform the CWS-Index for evaluating the expertise level of the experts as follows:

(28)

(29)

(30)

where

4.3. Comprehensive Ranking of Experts’ Expertise Level Under Multiple Criteria

Next, when the experts provide their pairwise comparison judgements for the mutual adjacent alternatives in fuzzy and multiplicative preference format under

criteria, we consider the comprehensive ranking method of their expertise level.

Let

express the expertise level of the expert

obtained under each criterion

from the adjacent pairwise comparison technique. Let

be the ranking by

Then, one can make ranking matrix for the expertise levels of experts,

In order to complete the comprehensive ranking for the expertise levels of experts, we can apply the Borda Count to the matrix

[29]

. This method ranks the expertise levels of the experts according to the total score of each expert with respect to the various criteria. For the entries of the matrix,

, the score of each expert,

is given as below:

(31)

The higher the score

, the higher is the expert’s ranking.

If the criteria have different relative importance weights, we get the synthetic expertise level of the expert

as below and rank according to their maginitude.

(32)

where

denote the importance weight of the criterion

and

hold.

One can apply the method of entropy to the matrix

to determine the objective weights of the criteria. Entropy method is well known as a kind of objective weighting methods to measure the usefull information of obtained data, the smaller the information entropy is, the bigger the weight of the criterion is

[28]

Let

be the vector of weights of the criteria obtained by the method of entropy.

is calculated as follows:

where

, shows the value of information entropy of the criterion

; and

, shows the standard expertise level of the expert

under the criterion

. Next, one can get the score

from the equation (32) and ranks the experts.

5. Weighting for Expertise Levels of Experts

Expertise-based ranking of experts can be applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM.

The vector the total scores for the expertise levels of the experts,

is regarded as the original data sequence. As CWS-indices to evaluate the expertise levels of the experts are formed of the ratios of the variances, they may have large differences. For which reason, to improve the smoothness of the original data sequence under the surcumstance of remaining its basic characteristic, we apply the following logarithmic function transformation to deal with the original data sequence.

where

. It is clear that

Let

be a vector of the experts’ importance weights.

is determined as follows:

Suppose that the decision matrices provided by the experts,

, are given, where

indicates the evaluation value of the attribute of the alternative

for the index

provided by the expert

. Then, we can obtain the aggregated decision matrix of the group,

, by considering the experts’ importance weights as follows:

If the vector of the importance weights of the criteria is given, we compute the synthetic score of the alternative

as follows:

The higher the value

, the greater is the importance of the alternative

6. Illustrative Example

In order to show the workability of the proposed method, we start with a numerical example to illustrate it. Suppose there are four people who are expert at judging the five senses on the wet goods. These experts are expressed as

. In order to judge the five senses on the wet goods, they ought to be assess four wet goods: alcoholic liquor, distilled liquor, brewage and fruit wine, with various sensible characteristics such as bouquet, taste and clearness. The wet goods form a set of four alternatives

. And the sensible characteristics form a set of three criteria

The experts were asked to fill their judgements in the blank space of Table 1, with the MPRs

represents the relative importance of the alternative

to the alternative

Table 1. The evaluation form.


1
	1
		1
			1

Suppose that the experts’ judgements under each criterion in the evaluation form of Table 1 are given as Table 2.

Table 2. The experts’ judgements under each criterion.



2	2	2	1	4	5	1	5	7	2	3	3
6	4	3	4	4	7	4	5	7	3	4	4
8	5	6	3	6	8	4	4	6	8	8	5
3	2	2	1	3	6	3	6	7	1	3	3
5	3	5	1	3	6	1	6	8	6	3	3
8	5	5	1	4	6	3	3	5	6	8	6

For example, from Table 2, the multiplicative judgement matrix of the expert

under the criterion

is formed as follows:

The estimation values of the entries of the matrix

are obtained from the equation (12) as Table 3.

Table 3. The estimation values of the entries of matrix

Before transformation			After transformation ( $\log_{9} a_{ij}^{(1)}$ )
Actual judgement	Estimation value		judgement	Estimation value
2	2	1.6	0.315	0.315	0.214
6	6	1	0.815	0.815	0
8	10	0.75	0.946	1.048	-0.131
3	3	0.625	0.5	0.5	-0.214
5	4	24	0.732	0.631	1.446
8	1.333	1.667	0.946	0.131	0.232

Applied the transformation procedure in subsection 3.1 to Table 3, one can get Table 4, and calculate CWS-index from the equations (8), (9), (10).

Table 4. The transformation values of the entries of matrix

A^{(1)}

	Transformation procedure in subsection 3.1					$\sum_{j = 1}^{n - 1} {(M_{ij} - M_{i})}^{2}$
	Judgement	Estimation value				$\sum_{j = 1}^{n - 1} {(M_{ij} - M_{i})}^{2}$
	1.615	1.615	1.384	1.538	1.397	0.036
	3.451	3.451	1.000	2.634	0.007	4.006
	4.211	4.913	0.820	3.315	0.354	9.584
	2.137	2.137	0.723	1.666	1.111	1.334
	3.043	2.608	9.000	4.883	4.680	25.514
	4.211	1.220	1.424	2.285	0.189	5.585
sum					7.74	46.05

As above, one can find the CWS-indices for every criterion and every expert as Table 5.

Table 5. CWS-indices and ranking of the experts under the criteria.

Result
CWS-index	1.21	0.984	1.678	0.812
Ranking
CWS-index	0.834	0.822	0.76	0.787
Ranking
CWS-index	0.748	2.225	0.767	0.686
Ranking

Table 5 shows that the experts

have the highest sensible ability in the criteria ‘bouquet’, ‘taste’, ‘clearness’, respectively. While the expert

has the lowest sensible ability in the criterion ‘taste’, the expert

in the criteria ‘bouquet’ and ‘clearness’ is so.

Next, we apply the adjacent pairwise comparison technique to the data of Table 2 to evaluate the experts’ sensible ability. For example, according to the discussion of subsection 4.2, the multiplicative judgement matrix of the experts’ group under the criterion

can be formed as follows:

Then, we can obtain the sensible ability of the experts under the criterion

as follows:

Download: Download full-size image

The CWS-Indices for all criteria are given as Table 6.

Table 6. CWS-indices and ranking using the adjacent pairwise comparison technique.

Result
CWS-index	2.225	0.874	2.094	0.283
Ranking
CWS-index	0.129	0.198	0.08	0.071
Ranking
CWS-index	0.108	0.223	0.185	0.087
Ranking

Compared Table 6 with Table 5, the experts have the same ranking in the criteria ‘clearness’, while there exist a little difference in the criteria ‘bouquet’, ‘taste’.

Now, according to Table 5 and Table 6, we form the evaluation matrices for the experts’ sensible ability,

, respectively; then we complete the comprehesive ranking for their sensible ability by the Borda Count in subsection 4.3.

Table 7. The comprehesive ranking by the Borda Count.

	Result
Table 5	(equation (31))	6	7	5	0
Table 5	Ranking
Table 6	(equation (31))	6	6	5	1
Table 6	Ranking

As seen in Table 7, the comprehensive ranking of the experts obtained by the adjacent pairwise comparison technique includes the result obtained by the traditional pairwise comparisons. This shows the acceptability of the adjacent pairwise comparison technique.

Now, we suppose that the pairwise comparison judgements of the experts for the alternative

under each criterion are unknown. For example, the incomplete multiplicative judgement matrix of the expert

under the criterion

is made from Table 2 as bellow.

Applied the algorithm 1 of subsection 3.3, we can estimate the missing elements of the matrix

. Refered to the Remark 1 in subsection 3.3, since

we can take

to ensure

. Therefore we have

. Similarly, one can estimate the incomplete multiplicative judgements of the experts for the alternative

under each criterion (Table 8).

Table 8. The estimation of the incomplete judgements of the experts for the alternative

X_{1}



1/3	1	1	1	1	1	2	1	1/2	1	1/2	1
1	2	2	1	3	6	6	6	7/2	1	3/2	3
29/6	13/2	15/2	3/2	11/2	9	7	6	21/4	9	19/4	15/2

According to the method in subsection 3.1, the CWS-Indices and ranking are obtained as Table 9.

Table 9. The CWS-Indices and ranking for the incomplete judgements.

Result
CWS-Index	5.51	2.581	4.737	0.2
Ranking
CWS-Index	2.132	2.282	0.728	3.386
Ranking
CWS-Index	4.254	5.977	2.742	1.877
Ranking

And we do the comprehensive ranking their sensible ability by Borda Count (Table 10).

Table 10. The comprehensive ranking by Borda Count.

Result
	6	6	3	3
Ranking

Obviously, the result of Table 10 includes the result of Table 7. This shows the acceptability of the method proposed in subsection 3.3.

7. Conclusion

This paper mainly studies on the method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats. First of all, we discuss on the combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations on the basis of the methodology proposed in

[3]

, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs under certain criterion. In particular, we develop an algorithm to estimate the missing elements of the incomplete MPR and show the properties supporting the algorithm. At the same time, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of the pairwise comparison judgements provided by the experts in GDM with multiple criteria. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Through the numerical analyses we confirm that proposed method is acceptable and can fill up some gaps in GDM. Further study may extend to incomplete preference relations under linguistic

[30]

, interval

[16]

, intuitionistic fuzzy

[33]

, hesitant fuzzy linguistic

[31]

and interval-valued Pythagorean fuzzy

[34]

, Fermatean fuzzy

[32]

environment in GDM.

Abbreviations

GDM	Group Decision Making
DM	Decision Maker
CWS-Index	Cochran-Weiss-Shanteau Index
AC	Additive Consistency
FPR	Fuzzy Preference Relation
MPR	Multiplicative Preference Relation
MCDM	Multi-Criteria Decision Making
PCM	Pairwise Comparison Matrix

Acknowledgments

The authors would like to thank the Editor-in Chief, Associate Editor and anonymous reviewers for their insightful and constructive commendations that have led to this improved version of the paper.

Author Contributions

SongHo Sim: Methodology, Validation

SinHyok Kim: Writing – original draft, Writing – review & editing

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	D. J. Weiss, J. Shanteau, Empirical assessment of expertise, Hum. Factors 45(1) (2003) 104–116.
[2]	J. Shanteau, D. J. Weiss, R. P. Thomas, J. C. Pounds, Performance-based assessment of expertise: how to decide if someone is an expert or not?, Eur. J. Oper. Res. 136(2) (2002) 253-263.
[3]	E. Herowati, U. Ciptomulyono, J. Parung, Suparno, Expertise-based ranking of experts: An assessment level approach, Fuzzy Sets Syst. 315 (2017) 44-56.
[4]	V. Malhotra, M. D. Lee, A. Khurana, Domain experts influence decision quality: towards a robust method for their identification, J. Pet. Sci. Eng. 57(1–2) (2007) 181–194.
[5]	Michael D. Lee, Mark Steyvers, M. d. Young, B. Miller, Inferring expertise in knowledge and prediction ranking tasks, Top. Cogn. Sci. 4(1) (2012) 151–163.
[6]	F. Chiclana, E. Herrera-Viedma, F. Herrera, S. Alonso, Some induced ordered weighted averaging operators and their use for solving group decision making problems based on fuzzy preference relations, Eur. J. Oper. Res. 182(1) (2007) 383–399.
[7]	E. Herrera-Viedma, F. Chiclana, F. Herrera, S. Alonso, Group decision-making model with incomplete fuzzy preference relations based on additive consistency, IEEE Trans. Syst. Man Cybern. B 37(1) (2007) 176–189.
[8]	E. Herrera-Viedma, F. Herrera, F. Chiclana, M. Luque, Some issues on consistency of fuzzy preference relations, Eur. J. Oper. Res. 154(1) (2004) 98–109.
[9]	S. Alonso, F. Chiclana, F. Herrera, E. Herrera-Viedma, J. Alcal’a-Fdez, C. Porcel, A consistency-based procedure to estimate missing pairwise preference values, Int. J. Intell. Syst. 23(2) (2008) 155–175.
[10]	S. Alonso, F. J. Cabrerizo, F. Chiclana, F. Herrera, E. Herrera-Viedma, Group decision making with incomplete fuzzy linguistic preference relations, Int. J. Intell. Syst. 24(2) (2009) 201–222.
[11]	G. Zhang, Y. Dong, Y. Xu, Linear optimization modeling of consistency issues in group decision making based on fuzzy preference relations, Expert Syst. Appl. 39(3) (2012) 2415–2420.
[12]	G. Kou, Y. Peng, X. R Chao, E. Herrera-Viedma, F. E. Alsaadi, A geometrical method for consensus building in GDM with incomplete heterogeneous preference information, Appl. Soft Comput. 105 (2021). https://doi.org/10.1016/j.asoc.2021.107224
[13]	E. Herowati, U. Ciptomulyono, J. Parung, Suparno, Expertise-based experts importance weights in adverse judgment, J. Eng. Appl. Sci. 9(9) (2014).
[14]	F. Chiclana, F. Herrera, E. Herrera-Viedma, Integrating three representation models in fuzzy multipurpose decision making based on fuzzy preference relations, Fuzzy Sets Syst. 97(1) (1998) 33–48.
[15]	L. Mikhailov, Deriving priorities from fuzzy pairwise comparison judgements, Fuzzy Sets Syst. 134 (2003) 365–385.
[16]	A. Khalid, I. Beg, Incomplete interval valued fuzzy preference relations, Inform. Sci. 348 (2016) 15–24.
[17]	S. P. Wan, D. F. Li, Fuzzy LINMAP approach to heterogeneous MADM considering comparisons of alternatives with hesitation degrees, Omega 41 (2013) 925–940.
[18]	Y. J. Xu, R. Patnayakuni, H. M. Wang, A method based on mean deviation for weight determination from fuzzy preference relations and multiplicative preference relations, Int. J. Inform. Technol. Decis. Mak. 11 (2012) 627–641.
[19]	Z. S. Xu, X. Q. Cai, S. S. Liu, Nonlinear programming model integrating different preference formats, IEEE Trans. Syst. Man Cybern. A 41 (2011) 169–177.
[20]	E. Herrera-Viedma, F. Herrera, F. Chiclana, A consensus model for multiperson decision-making with different preference formats, IEEE Trans. Syst. Man, Cybern. A 32 (2002) 394–402.
[21]	P. Xie, J. Wu, H. Du, The relative importance of competition to contagion: evidence from the digital currency market, Financial Innov. 5(1) (2019) 41.
[22]	F. Y. Meng, X. Chen, An approach to incomplete multiplicative preference relations and its application in GDM, Inform. Sci. 309 (2015) 119–137.
[23]	G. Kou, D. Ergu, C. S. Lin, Y. Chen, Pairwise comparison matrix in multiple criteria decision making, Technol. Econ. Dev. Econ. 22 (2016) 738–765.
[24]	T. L. Saaty, The Analytic Hierarchy Process, McGraw-Hill, New York, USA, 1980.
[25]	F. Herrera, E. Herrera-Viedma, F. Chiclana, Multiperson decision-making based on multiplicative preference relations, European J. Oper. Res. 129 (2001) 372–385.
[26]	A. O. Efe, Utility representation of an incomplete preference relation. J. Econ Theory, 104 (2002) 429-449.
[27]	F. Herrera et al, A consensus model for group decision making with incomplete fuzzy preference relations. IEEE Trans. Syst. 15(5) (2007) 863–877.
[28]	A. Shemshadi, H. Shirazi, M. Toreihi, M. J. Tarokh, A fuzzy VIKOR method for supplier selection based on entropy measure for objective weighting. Expert Syst. Appl. 38 (2011) 12160–12167.
[29]	Z. Mahdi, S. Ferenc, Multicriteria Analysis: Application to water and environment management. Springer, Verlag Berlin Heidelberg, 2011.
[30]	D. Cheng, Z. Zhou, F. Cheng, J. Wang, Deriving heterogeneous experts weights from incomplete linguistic preference relations based on uninorm consistency, Knowl. Based Syst. 150 (2018) 150–165.
[31]	C. Li, R. M. Rodríguez, L. Martínez, Y. C. Dong, F. Herrera, Consistency of hesitant fuzzy linguistic preference relations: An interval consistency index, Inform. Sci. 432 (2018) 347–361.
[32]	Senapati T, Yager RR, Fermatean fuzzy sets. J Amb Intell Hum Comput 11(2) (2020) 663–674.
[33]	R. Wang, Y. L. Li, A novel approach for group decision-making from intuitionistic fuzzy preference relations and intuitionistic multiplicative preference relations. Inform. 9(55) (2018); https://doi.org/10.3390/info9030055
[34]	H. M. Li, Y. C. Cao, L. M. Su, Q. Xia, An interval Pythagorean fuzzy multi-criteria decision making method based on similarity measures and connection numbers. Inform. 10(80) (2019); https://doi.org/10.3390/info10020080

Cite This Article

Plain Text BibTeX RIS

APA Style

Sim, S., Kim, S. (2025). Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats. American Journal of Engineering and Technology Management, 10(5), 69-83. https://doi.org/10.11648/j.ajetm.20251005.11

Copy | Download

ACS Style

Sim, S.; Kim, S. Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats. Am. J. Eng. Technol. Manag. 2025, 10(5), 69-83. doi: 10.11648/j.ajetm.20251005.11

Copy | Download

AMA Style

Sim S, Kim S. Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats. Am J Eng Technol Manag. 2025;10(5):69-83. doi: 10.11648/j.ajetm.20251005.11

Copy | Download

@article{10.11648/j.ajetm.20251005.11,
  author = {SongHo Sim and SinHyok Kim},
  title = {Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats
},
  journal = {American Journal of Engineering and Technology Management},
  volume = {10},
  number = {5},
  pages = {69-83},
  doi = {10.11648/j.ajetm.20251005.11},
  url = {https://doi.org/10.11648/j.ajetm.20251005.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajetm.20251005.11},
  abstract = {The group decision-making (GDM) is a process of aggregating the preference information of each decision maker in a group into a collective opinion under certain decision criteria. When a decision is made by decision makers (DMs), the group decision or group opinion is a result of the integration of the individual opinions by a mathematical aggregation. An important thing in the integration of the individual opinions is which DMs’ opinions should be of importance in the aggregation process. The group opinion is heavily influenced by the level of expertise of the DMs. In real-life group decision making (GDM) problems, preference relations given by decision makers (DMs) are often heterogeneous because of their expertise and different decision habits. The expertise level of the DMs can be estimated using the ‘the ability to differentiate consistently’, a ratio between discrimination and inconsistency. Each expert has different expertise in different criteria and has his/her limited capacity in constructing of pairwise comparison preference relations. In this paper, we focus on a method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats. The combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations is discussed, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs, including FPRs under certain criterion. Then, since the experts have the limited capacity in constructing of pairwise comparison preference relations in GDM with multiple criteria, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of judgements. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Numerical analyses show that proposed method is simple than the previous methods and can fill up some gaps in GDM.
},
 year = {2025}
}

Copy | Download

TY  - JOUR
T1  - Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats

AU  - SongHo Sim
AU  - SinHyok Kim
Y1  - 2025/10/31
PY  - 2025
N1  - https://doi.org/10.11648/j.ajetm.20251005.11
DO  - 10.11648/j.ajetm.20251005.11
T2  - American Journal of Engineering and Technology Management
JF  - American Journal of Engineering and Technology Management
JO  - American Journal of Engineering and Technology Management
SP  - 69
EP  - 83
PB  - Science Publishing Group
SN  - 2575-1441
UR  - https://doi.org/10.11648/j.ajetm.20251005.11
AB  - The group decision-making (GDM) is a process of aggregating the preference information of each decision maker in a group into a collective opinion under certain decision criteria. When a decision is made by decision makers (DMs), the group decision or group opinion is a result of the integration of the individual opinions by a mathematical aggregation. An important thing in the integration of the individual opinions is which DMs’ opinions should be of importance in the aggregation process. The group opinion is heavily influenced by the level of expertise of the DMs. In real-life group decision making (GDM) problems, preference relations given by decision makers (DMs) are often heterogeneous because of their expertise and different decision habits. The expertise level of the DMs can be estimated using the ‘the ability to differentiate consistently’, a ratio between discrimination and inconsistency. Each expert has different expertise in different criteria and has his/her limited capacity in constructing of pairwise comparison preference relations. In this paper, we focus on a method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats. The combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations is discussed, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs, including FPRs under certain criterion. Then, since the experts have the limited capacity in constructing of pairwise comparison preference relations in GDM with multiple criteria, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of judgements. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Numerical analyses show that proposed method is simple than the previous methods and can fill up some gaps in GDM.

VL  - 10
IS  - 5
ER  -

Copy | Download

Author Information

SongHo Sim

Institute of Mathematics, State Academy of Sciences, Pyongyang, DPR Korea

Contact Email

http://orcid.org/0009-0005-0894-4596
SinHyok Kim

Institute of Mathematics, State Academy of Sciences, Pyongyang, DPR Korea

Download PDF

Submit an Article

Plain Text BibTeX RIS

APA Style

Sim, S., Kim, S. (2025). Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats. American Journal of Engineering and Technology Management, 10(5), 69-83. https://doi.org/10.11648/j.ajetm.20251005.11

Copy | Download

ACS Style

Sim, S.; Kim, S. Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats. Am. J. Eng. Technol. Manag. 2025, 10(5), 69-83. doi: 10.11648/j.ajetm.20251005.11

Copy | Download

AMA Style

Sim S, Kim S. Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats. Am J Eng Technol Manag. 2025;10(5):69-83. doi: 10.11648/j.ajetm.20251005.11

Copy | Download

@article{10.11648/j.ajetm.20251005.11,
  author = {SongHo Sim and SinHyok Kim},
  title = {Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats
},
  journal = {American Journal of Engineering and Technology Management},
  volume = {10},
  number = {5},
  pages = {69-83},
  doi = {10.11648/j.ajetm.20251005.11},
  url = {https://doi.org/10.11648/j.ajetm.20251005.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajetm.20251005.11},
  abstract = {The group decision-making (GDM) is a process of aggregating the preference information of each decision maker in a group into a collective opinion under certain decision criteria. When a decision is made by decision makers (DMs), the group decision or group opinion is a result of the integration of the individual opinions by a mathematical aggregation. An important thing in the integration of the individual opinions is which DMs’ opinions should be of importance in the aggregation process. The group opinion is heavily influenced by the level of expertise of the DMs. In real-life group decision making (GDM) problems, preference relations given by decision makers (DMs) are often heterogeneous because of their expertise and different decision habits. The expertise level of the DMs can be estimated using the ‘the ability to differentiate consistently’, a ratio between discrimination and inconsistency. Each expert has different expertise in different criteria and has his/her limited capacity in constructing of pairwise comparison preference relations. In this paper, we focus on a method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats. The combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations is discussed, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs, including FPRs under certain criterion. Then, since the experts have the limited capacity in constructing of pairwise comparison preference relations in GDM with multiple criteria, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of judgements. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Numerical analyses show that proposed method is simple than the previous methods and can fill up some gaps in GDM.
},
 year = {2025}
}

Copy | Download

TY  - JOUR
T1  - Expertise-Based Ranking of Experts in Multicriteria Group Decision Making with Different Preference Representation Formats

AU  - SongHo Sim
AU  - SinHyok Kim
Y1  - 2025/10/31
PY  - 2025
N1  - https://doi.org/10.11648/j.ajetm.20251005.11
DO  - 10.11648/j.ajetm.20251005.11
T2  - American Journal of Engineering and Technology Management
JF  - American Journal of Engineering and Technology Management
JO  - American Journal of Engineering and Technology Management
SP  - 69
EP  - 83
PB  - Science Publishing Group
SN  - 2575-1441
UR  - https://doi.org/10.11648/j.ajetm.20251005.11
AB  - The group decision-making (GDM) is a process of aggregating the preference information of each decision maker in a group into a collective opinion under certain decision criteria. When a decision is made by decision makers (DMs), the group decision or group opinion is a result of the integration of the individual opinions by a mathematical aggregation. An important thing in the integration of the individual opinions is which DMs’ opinions should be of importance in the aggregation process. The group opinion is heavily influenced by the level of expertise of the DMs. In real-life group decision making (GDM) problems, preference relations given by decision makers (DMs) are often heterogeneous because of their expertise and different decision habits. The expertise level of the DMs can be estimated using the ‘the ability to differentiate consistently’, a ratio between discrimination and inconsistency. Each expert has different expertise in different criteria and has his/her limited capacity in constructing of pairwise comparison preference relations. In this paper, we focus on a method to assess expertise-based ranking of experts in GDM with multiple criteria and different preference representation formats. The combination of expertise as ‘the ability to differentiate consistently’ and the consistency of various types of preference relations is discussed, when experts give their judgements for alternatives in various types of preference relations, such as multiplicative preference relations (MPRs), utility values, preference orderings, even incomplete FPRs or MPRs, including FPRs under certain criterion. Then, since the experts have the limited capacity in constructing of pairwise comparison preference relations in GDM with multiple criteria, we propose a expertise-based ranking method using the adjacent pairwise comparison technique in order to reduce the number of judgements. Finally, expertise-based ranking of experts is applied to aggregate the individual opinions into a group opinion by developing into experts’ importance weights in GDM. Numerical analyses show that proposed method is simple than the previous methods and can fill up some gaps in GDM.

VL  - 10
IS  - 5
ER  -

Copy | Download



2	2	2	1	4	5	1	5	7	2	3	3
6	4	3	4	4	7	4	5	7	3	4	4
8	5	6	3	6	8	4	4	6	8	8	5
3	2	2	1	3	6	3	6	7	1	3	3
5	3	5	1	3	6	1	6	8	6	3	3
8	5	5	1	4	6	3	3	5	6	8	6



2	2	2	1	4	5	1	5	7	2	3	3
6	4	3	4	4	7	4	5	7	3	4	4
8	5	6	3	6	8	4	4	6	8	8	5
3	2	2	1	3	6	3	6	7	1	3	3
5	3	5	1	3	6	1	6	8	6	3	3
8	5	5	1	4	6	3	3	5	6	8	6



2	2	2	1	4	5	1	5	7	2	3	3
6	4	3	4	4	7	4	5	7	3	4	4
8	5	6	3	6	8	4	4	6	8	8	5
3	2	2	1	3	6	3	6	7	1	3	3
5	3	5	1	3	6	1	6	8	6	3	3
8	5	5	1	4	6	3	3	5	6	8	6