Generic Skills Measurement of Students at Fukuoka Dental College: The Usefulness of the Progress Report on Generic Skills (PROG) Test

Abstract

In recent years, university education has been called upon to cultivate new abilities known as “generic skills,” which have been defined in various ways by the different ministries and agencies advocating their necessity. These generic skills are also indispensable for dentistry graduates and skills training is important in the context of dental education. However, while conventional dental education has heretofore conducted assessments with regard to knowledge, understanding and professional ability when evaluating learning results, higherorder integrative capabilities such as generic skills have not been evaluated. Therefore, after introducing the Progress Report on Generic Skills (PROG) Test as an evaluation method for measuring generic skills to examine their relationship with academic achievement. Administering PROG Test soon after the time of admission, it was possible to measure generic skills at the time of admission, within which the overall literacy results were related to academic ability at the time of admission and subsequent academic performance. We can report the suggested possibility that the PROG test may be used to evaluate generic skills among dentistry students.

Keywords

Generic skills, PROG test, Evaluation, Dentistry students

Introduction

In recent years, university education has been called upon to cultivate new abilities known as “generic skills”. Generic skills, also known as “translatable skills,” are high-order skills that can be applied in a variety of circumstances, such as creativity, flexibility, independence, ability to work in a team, communicative ability, critical thinking, time management, leadership, planning ability, and self-management [1]. These skills have been variously defined by the different Japanese government ministries and agencies advocating their necessity, as with the term “human power” proposed by the Cabinet Office [2], “core skills for working adults” proposed by the Ministry of Economy, Trade and Industry (METI) [3], “academic competency” proposed by the Central Education Council [4], “employability” proposed by the Ministry of Education, Culture, Sports, Science and Technology (MEXT) [5], and “fundamental general-purpose skills” in the field of career education [6].

Dental education requires not only the interpretive and thinking abilities that accompany the simple memories gradually cultivated from early education through high school but also the general-purpose skills such as clinical reasoning and the integration of learning content from individual subjects taken thus far; these skills are also indispensable abilities for dentists to have upon graduation. Although Kawashima describes generic skills training by such means as Project-Based Learning internships and service learning [1], METI favors skills training in programs related to university specializations over special programs aiming solely at generic skills training [7]. However, very few universities carry out generic skills training over the entire undergraduate curriculum, with most undertaking such training in the context of first-year introductory and liberal arts subjects [8]. At Fukuoka Dental College, we have introduced an initiative that aims to carry out generic skills training in our first-year introductory and liberal arts subjects as well as in the entire six-year undergraduate curriculum, and then evaluate and visualize the skills that have been acquired. However, while conventional dental education has heretofore conducted assessments with regard to knowledge, understanding, and professional ability when evaluating learning results, higher-order integrative capabilities such as generic skills have not been evaluated.

Therefore, in the academic year 2016, we introduced the Progress Report on Generic Skills (PROG) Test to measure generic skills, and have examined the actual extent and nature of the relationship between generic skills at the time of admission, academic ability at the time of admission and Grade Point Average (GPA) in Year 1.

Target and Methods

Target

In the beginning, the survey was explained orally and via a document, and after checking the consent, it was conducted.

1. When it is used for any purpose other than study support, it becomes anonymous to connect all data and process and use it so that an individual cannot specify.

2. You can refuse use of data.

3. Opt-out is possible, even if it once agrees.

Get consent for use of education data from students and parents in research, upon admission than 2018. The period before it, after explaining the above mentioned contents to a student, was put into effect. Although all the target students' consent was checked orally, a less than 20-year-old guardian's consent is not checking this time. Thus, this research was done by having adopted the principle of Declaration of Helsinki, and was performed by obtaining the approval of Ethical Review Board, Fukuoka Dental College.

The study involved 97 students, all of whom were students enrolled in Year 1 in the academic year 2016, and who appeared for the Comprehensive Academic Achievement Examination (CAAE) and PROG Test conducted on April 12, 2016. The competency elements of the PROG Test results for 94 of these students were used, after 3 of the 97 students were identified as ineligible due to an insufficient number of completed responses.

Analysis of PROG Test Results

Tables 1 and 2 shows an outline of the PROG Test. For the literacy test, we used a 7-stage judgment result for overall literacy, and for the 6 items comprising 4 problem-solving skills and 2 processing skills, we used a 5-stage judgment result. For the competency test, we used a 7-stage judgment result for the overall competency and 3 competency skills, and a 5-stage judgment result for the 9 constituent elements comprising these 3 competencies.

Table 1. PROG (progress report on generic skills) test outline.

Table 2. Evaluation scores and level settings for the PROG Test.

Data Used

We used the total scores for the PROG Test and the CAAE administered on the same day, students’ Year 1 GPA for 2016 and their grades for 22 subjects offered in Year 1. After linking this data using the PROG Test results and student identification numbers, the student identification numbers were deleted so that the data could be used as unlinkable anonymized data.

Statistical Processing

To investigate the relationships among the PROG Test results, academic ability at the time of admission, and learning results after admission, we calculated Spearman’s rank correlation coefficient with respect to the judgment result of each element of the PROG Test and the overall CAAE scores, Year 1 GPA scores, and grades for 22 subjects offered in Year 1.

Results

The averages of the results for overall CAAE scores, Year 1 GPA, grades for subjects offered in Year 1, and PROG Test measurement items are shown in Tables 3-5.

Table 3. Total comprehensive academic achievement exam (CAAE) scores and mean GPA for year 1.

Table 4. Mean scores for subjects offered in year 1.

Table 5. Mean scores for PROG test measurement items.

Correlations among overall CAAE Scores, Year 1 GPA, and PROG Test Measurement Items (Table 6).

Table 6. Correlations among total CAAE scores, year 1 GPA and PROG test measurement items.

Relationships between overall CAAE scores and PROG test measurement items: While a significant positive correlation was found between overall CAAE scores and overall literacy (ρs=0.303, p<0.01), no significant correlation was found with overall competency. In terms of literacy elements, significant positive correlations were found with the ability to analyze data, the ability to identify problems, and verbal processing ability (data analysis: ρs=0.300, p<0.01, ability to identify problems: ρs=0.245, p<0.05, verbal processing ability: ρs=0.202, p<0.05). In terms of competency elements, no significant correlations were found with overall CAAE scores for any items.

Correlations between scores for subjects offered in year 1 and PROG test evaluation items (Tables 7-9).

Table 7. Correlations between scores for subjects offered in year 1 and PROG test evaluation items.

Table 8. Correlations between scores for subjects offered in year 1 and PROG test evaluation items.

Table 9. Correlations between scores for subjects offered in year 1 and PROG test evaluation items.

Relationships between grades in subjects offered in year 1 and overall literacy/overall competency for PROG test measurement items: Among the 22 subjects offered in Year 1, while all but 3 subjects (“Long-Term Care Facility/Brushing Experience,” “Data Processing Exercises,” “Data Processing Exercises II”) were found to have a significant positive correlation with overall literacy, only the “Debate Exercise” course was found to have a significant positive correlation with overall competency (ρs=0.257, p<0.05).

Relationships between grades in subjects offered in year 1 and PROG test measurement items literacy elements (4 Problem- Solving Skills and 2 Processing Skills): Among the 22 subjects offered in Year 1, significant positive correlations were found with the literacy element of data collection in 10 subjects, of data analysis in 10 subjects, of the ability to identify problems in 17 subjects, of conceptual ability in 9 subjects, of verbal processing ability in 4 subjects, and of non-verbal processing ability in 2 subjects. Looking at each course in turn, significant positive correlations were found with 5 elements in 2 subjects, 4 elements in 3 subjects, 3 elements in 3 subjects, 2 elements in 8 subjects, and 1 element in 5 subjects, with only 1 course in which no significant correlations were observed with any element.

Relationships between grades in subjects offered in year 1 and PROG test measurement items competency elements (3 Skills): Among the 22 subjects offered in Year 1, correlations were found with the 3 subjects of “Basic Science and Physics,” “Practical English II,” and “Debate Exercise”; significant positive correlations in Basic Science and Physics were observed with regard to the two skill sets of Basic Self-Management Skills (ρs=0.284, p<0.05) and Basic Problem-Oriented Skills (ρs=0.342, p<0.01), in Practical English II with regard to the skill set of Basic Self-Management Skills (ρs=0.222, p<0.05), and in Debate Exercise with regard to the two skill sets of Basic Interpersonal Skills (ρs=0.214, p<0.05) and Basic Self-Management Skills (ρs=0.286, p<0.01). The relationship between grades for these 3 subjects and the 9 constituent competency elements of these 3 skill sets is shown in Table 10. Significant positive correlations in Basic Science and Physics were observed with the two elements of emotional control (ρs=0.361, p<0.01) and the ability to identify problems (ρs=0.381, p<0.01), in Practical English II with the three elements of Leadership (ρs=0.207, p<0.05), creative confidence (ρs=0.244, p<0.05), and perseverance (ρs=0.215, p<0.05) and in Debate Exercise with the four elements of leadership (ρs=0.270, p<0.01), creative confidence (ρs=0.305, p<0.01), ability to identify problems (ρs=0.215, p<0.05), and practical ability (ρs=0.256, p<0.05).

Table 10. Correlations between competency elements (9 Elements) and scores for 3 subjects found to have a significant positive correlation with competency elements (3 Skills).

Discussion and Conclusion

The PROG report used in this investigation is an assessment program, jointly developed by the Kawaijuku Educational Institution and RIASEC Inc., that supports the growth of general-purpose proficiencies, attitudes, and inclinations (generic skills) required by society. The PROG Test is an assessment test that objectively measures the skills required by society in terms of the two aspects of literacy and competency [7-10]. It has been used in various universities to measure learning outcomes with a view of educational improvement, as well as of encouraging student growth and subjective learning. Since its release in April, 2012, the PROG Test has already been administered at approximately 290 universities across Japan and taken by more than a quarter of a million students [11]. With the PROG Test, it is possible to objectively grasp the current status of individual students with measurements from the two standpoints of literacy and competency.

Using the items shown in Table 1, with regard to literacy, we check the skills to solve challenges by using knowledge of new problems and problems that have not previously been experienced, and with regard to competency, the behavioral styles and decision-making characteristics acquired to better cope with contextual circumstances [9,10]. Evaluation scores and level setting are shown in Table 2. For the evaluation of overall literacy, Stages 1 to 4 are set at the level achieved in the first year of study, with Stage 7 set at the level of someone who has completed an undergraduate program. For the evaluation of overall competency, Stages 1 to 4 are set at the level of someone who has completed an undergraduate program, while Stages 5 and 6 are set at the level expected for graduates who have been in the workforce for up to three years, and Stage 7 is set at the level of someone playing a leading role as a young worker [11].

Narita has pointed out that while a clear positive correlation is apparent in the relationship between PROG Test results and the Standard Deviation (SD) value for entrance examinations, in the sense that higher entrance examination SD values are associated with higher literacy scores, the relationship with regard to competency does not appear to be as strong as that with literacy; in terms of overall competency, very little difference was found among the High-SD Group (SD ≥ 55), Medium-SD Group (SD ≥ 45<55), and Low-SD Group (SD value< 45) [12]. Also, Yamamoto and Matsumoto state that basic academic ability up to high school is a necessary condition for literacy [13]. In the PROG Test results for first-year Fukuoka Dental College students in 2016 as well, while a significant positive correlation with overall literacy was found with overall CAAE scores, which measure basic academic ability at the time of admission, no significant correlation was found with respect to overall competency. Looking at the specific literacy elements with which correlations were found, we find that significant positive correlations were observed with respect to data analysis, the ability to identify problems, and verbal processing ability.

Yamamoto discussed the relationship between PROG Test results and GPA scores, reporting that test results do not necessarily correlate with GPA [14,15]. However, we found a correlation with GPA in the PROG Test results for first-year Fukuoka Dental College students in 2016. In the measurement items, while a significant positive correlation was found with respect to overall literacy, no significant correlation was found with respect to overall competency. Looking at the specific literacy elements with which correlations were found, we find that significant positive correlations were observed with respect to data analysis and the ability to identify problems. In terms of the correlation with GPA, given that the subjects offered in Year 1 at our school promote the development of basic academic ability and include several remedial education subjects in mathematics and science for advancing students to the next stage, it seems possible that this correlation may be due to the fact that basic academic ability at the time of admission is reflected in Year 1 GPA scores.

This study identified that by administering the PROG Test soon after the time of admission, it was possible to measure generic skills at the time of admission, within which the overall literacy results were related to academic ability at the time of admission and the Year 1 GPA scores after the first year of study. In addition, there is a possibility that this may also be related to subsequent academic performance, suggesting its usefulness for the instruction of individual students. In terms of the competency aspect of generic skills, no significant correlations were observed among PROG Test results, CAAE scores, and Year 1 GPA. However, the ability to respond well to contextual circumstances by taking advantage of the behavioural styles and decision-making characteristics acquired measured from this competency standpoint also represent an indispensable capability for engaging in dentistry. Yamamoto states that while university education does foster competency, its evaluation has been lacking such that the assessment of required subjects and their results while being cognizant of generic skills training remains a challenge [13]. Among the subjects offered in Year 1 at our school, there were some subjects in which a significant positive correlation was found between grade scores and competency-related measurement items. These subjects included observational evaluation such as interpersonal dialog and group work performance, and while this seems to be due to consistency between evaluation items and competencyrelated measurement items, this needs to be studied further in the future. Also, as reported by Sasakawa [16,17] and Saga et al. [18]. It appeared that it would also be possible to evaluate the growth of students’ generic skills by administering the PROG Test again after a period of several years and comparing the results.

Acknowledgments

This study was conducted with the support of the MEXT Acceleration Program for University Education Rebuilding. It should also be noted in relation to this presentation that there are no company-related conflicts of interest to disclose.

References

1. Kawashima T. International trends in generic skills assessment an international comparative study of outcomes evaluation in undergraduate education and generic skills training, 2007-2009. Grant-in-Aid for Scientific Research (B) Report of Study Findings. 2010;155-160.
2. http://www5.cao.go.jp/keizai1/2004/ningenryoku/0410houkoku.pdf
3. http://www.meti.go.jp/policy/kisoryoku/chukanhon.pdf
4. MEXT Central Education Council. Toward the construction of a bachelor’s degree curriculum. Official report, 2008.
5. http://www.mext.go.jp/b_menu/shingi/chukyo/chukyo4/029/siryo/1298671.htm
6. http://www.nier.go.jp/shido/centerhp/23career_shiryoushu/all_ver.pdf
7. METI Start today to foster and evaluate core skills for working adults: A wellspring of youth to support the Japan of tomorrow! 2007.
8. Hamana A. Generic skills training and outcomes evaluation. An International comparative study of outcomes evaluation in undergraduate education and generic skills training, 2007‒2009. Grant-in-Aid for Scientific Research (B) Report of Study Findings.2010;1-17.
9. http://www.kawai-juku.ac.jp/prog/
10. http://www.riasec.co.jp/prog_hp/
11. PROG seminar 2015. The image of students as seen from PROG. Kawaijuku Educational Institution and RIASEC Inc. 2015.
12. Narita H. The reconstruction of higher education based on the evidence (Pluralistic assessment of master of achievement). J Japan SocInform Knowledge 2014;24:393-403.
13. Yamamoto K and Matsumoto K. Using the PROG test to measure and train generic skills in first-year composition subjects. Law Review of the Law Society of Kyushu International University. 2013;19:51-62.
14. Yamamoto K. Generic skills training and outcomes evaluation (PROG Seminar Report). Kawaijuku Educational Institution and RIASEC Inc.2011;23-26.
15. Yamamoto K. Generic skills training and evaluation to elicit learning awareness among university students (PROG Seminar Report). Kawaijuku Educational Institution and RIASEC Inc. 2013;77-62.
16. Sasakawa A. A literacy-enhancement analysis using questions from the PROG test. Nagasaki University J Bus Eco. 2015;95:239-253.
17. Sasakawa A. The analysis of skill development among students using the PROG test. Annual Bulletin of the Nagasaki University. 2015;31:1-23.
18. Saga S, et al. Practices and evaluation of product discovery at University of Tsukuba. Proceedings of the Annual Meeting of the Japanese Society for Software Science. 2015;32:5.