IREX

Lead partner Department of computer technologies of teaching, National Medical University/Testing Board
Project director Irina Ye.Bulakh
Supporting partner Academy of Management and Informatics, Bielsko-Biala, Poland
Project director Franciszek Marecki

Irina Ye.Bulakh
Lead Partner Project Director

Project Dates May 1, 2000 - March 31, 2001

The project "A new instrument of curriculum development" was executed by Ukrainian - Polish partnership which joined three partners: Department of Medical Informatics and Computer Technologies of Training (National Medical University) , Testing Board of professional competence, affiliated with the Ministry of Public Health of Ukraine and Academy of Management and Informatics (Bielsko-Biala, Poland).

The project aimed at supporting the reform of higher education in Ukraine by means of validated instrument and technology for curriculum development. To meet the goal following objectives for the project have been formulated:

• studying of software instrument for curriculum development
• dissemination efforts and piloting web-based access to the instrument if the results of study are positive.

It was expected to achieve the objectives through a range of activities, including locating and obtaining of software instrument, its multifaceted investigation, obtaining of professional standards, preparation of descriptive information, presentations, leading workshops, publication of study results, piloting web-access to the instrument. During project new activities have been added in order to optimize the project and meet project terms. These new activities include development of the own software instrument, developing of standards for higher medical education that could be used as a material for study in the project, working on methodology of curriculum development which was not sufficiently specified by regulatory documents.

The goal of the project has been achieved.

Following results have been obtained during the project:

• software instrument for curriculum development has been created and investigated
• this instrument has been made accessible for health-related education and some other types of professional education (technical and agriculture)
• professional standards (OKH) for Medicine have been developed and approved by authorities
• national professional curriculum (OPP) is in process of development
• presentation of the instrument has been done at least during 12 events, including 4 structured workshops
• 18 publications related to standard development have been prepared
• the fundamental book "Diagnostics of higher education quality" is in print with one chapter fully devoted to standards in higher education

The work will continue beyond the project because of high interest of project partners to the topic, respect, expertise and knowledge about standards and their development obtained by partners with the help of the project.

Major Accomplishments (Were your original goals achieved?)

The project "A new instrument of curriculum development" was carried out by Ukrainian-Polish partnership. Partnership was formed by the following organizations: Department of Medical Informatics and Computer Technologies of Teaching at National Medical University (Kyiv, Ukraine), Testing Board of Professional Competence affiliated with the Ministry of Public Health of Ukraine and Academy of Management and Informatics (Bielsko-Biala, Poland). Additionally, the experts from University of Dundee Center for Medical Education (Scotland, UK) were involved in the project.

The project goal was to facilitate the reform of higher education in Ukraine through availability of valid instrument and technology of curriculum development. It was planned to ensure finding of such an instrument for curriculum development and administrating of its study, dissemination of information about such instrument to the target audience and start-up work for provision of the instrument through the Web. The “instrument of curriculum development” has wide meaning of a software tool (or conglomerate of software tools) that helps in development of educational standards (which in Ukraine should consist of three components: professional description - OKH, national curriculum - OPP and assessment tools).

The actuality of the goal was proved by the on-going educational reform of in Ukraine. This reform sought to transform the higher education in Ukraine to a standard-based (outcome-based) education by development and introduction into all areas of higher and professional education of new State Standards for Higher Education (constructed of three above-mentioned components). The process has been supported by relevant regulatory basis from Cabinet of Ministers and Ministry of Education.

The following may be named as desired results of the project:

• availability of evaluated curriculum development instrument and
• awareness of Ukrainian and international circles regarding project efforts in curriculum development through intensive dissemination policy.

Evaluated curriculum development instrument. Instrument for professional description (OKH). The project started with the assumption that three components of State Standards for Higher Education (SSHE) that are being developed in Ukraine can be supported with some separate software instruments. During project period we planned to study the tool which should help generation of professional description (OKH) only. By the beginning of the project such software tool was already applied by Testing Board in medical education and we intended to widen it for other professional areas. This software product represents the Access-based program that prompts expert to enter elements of professional description (OKH) in accordance with regulated structure. This first software product for “OKH” entering has been evaluated for its effectiveness and convenience by project staff (M.Mrouga, Y.Bogachkov, I.Filonchuk) and by invited specialty experts N.Khomchenkova. Evaluation of the instrument was executed with the help of higher medical educational standards (professional description - OKH) from Ukraine, Canada and United Kingdom. Evaluation showed that first version of the instrument allowed development of professional description (OKH) (1st part of SSHE) according to Ukrainian regulations, but it is very rigidly oriented towards the structure of Ukrainian standards and has semantic limitations which prevent fully automatic generation of standards.

Search for the instrument of curriculum design. Evaluation of software for constructing professional description (OKH) and interviewing of participation of standard development meeting brought understanding that development of instrument for generating of professional description (OKH) alone is neither reasonable nor interesting for the target audience. Only complex approach to standard development is valid (which foresees work on different Standards components). Therefore, there appeared a need for more complex software product. The worldwide search for available instruments (by means of Internet, educational and assessment listservs ASSESS and DRED, information from partners in Ukraine and abroad) returned several software programs. Few programs have been attractive for the project. Negotiations with the owners regarding conditions of software obtaining or purchase were undertaken but none of the program was purchased due to the number of reasons (Attachment #1).

Development of own software. Starting from the 3rd quarter of the project, project team started the programming of totally new, own complex software program for curriculum design (chief programmer - L.Rusina). The main task for the instrument was to allow generation of the curriculum (professional program) basing on provided professional description (OKH) and professional knowledge base.

Specialty area taken as an example was Medicine and other health-related specialties. The choice of specialty was defined by professional interests of Testing Board and Department of Medical Informatics, our involvement in Standards development for health professions and possibility to mobilize necessary number of Ukrainian specialty experts for the work as well as our wide international connections in medicine. We had possibility to use draft versions of standards for Medicine. In other professional areas no published standards in Ukraine were located. As we were not able to locate informatics standard in Ukraine, we did not evaluate any international non-medical standard as well.

In order to make programming possible, the project team was forced to develop the methodic of converting professional description (OKH) into curriculum. Now underlying the instrument is a concept that curriculum is a function of professional description (OKH) and professional knowledge base. Professional description (OKH) is presented in the form of skills that should be possessed by any given graduate. At the same time, skills should ground on knowledge that also should be specified and linked with definite skills. By that time, the professional description (OKH) for Medicine has already been created (in draft). The list of medical knowledge has been formulated by joint efforts of experts from different medical schools during last 4-5 years in the process of creating of medical licensing examinations for medical doctors. The details of this concept (OPP technology) have been worked out by the project team and applied for medical education (as a pilot area).

The task of the instrument is to allow establishing links between final skills that should be possessed by graduate, corresponding knowledge that should be mastered during period of study and teaching blocks (eg. subjects or modules) that should cover these knowledge and skills.

Evaluation. Curriculum-designing instrument has been evaluated in 3rd and 4th quarters of the project. Evaluation of the software product still continues by external experts and product may be additionally elaborated based upon expert conclusion. The evaluation of the program was administered by following methods:

• Direct work of project experts (N.Khomchenkova, N.Zelenina) with software. Experts assessed software convenience, limitations of beta-version, recommendations for software improvement.
• Assessment by project staff (I.Bulakh, I.Filonchuk, M.Mrouga). Staff assessed functionality and program interface.
• External evaluation (in process). This evaluation is done by laboratory of higher education at Donetsk Sate Medical University (laboratory chief - M.Garina), Research Center for Higher Education (MOE) (head of the nursing and agriculture department - K.Ohrimenko, head of medical and biological department - V.Gorchakov, head of technical department - V.Kuznetsov). The purpose of this evaluation is to assess validity of national curriculum (OPP) produced with the help of software. The design of constructed curriculum will be assessed through expert conclusions.
• The concept that is underlying the software has been discussed and assessed by expert from Centre for Medical Education Dr.M.F.Ben-David during her visit to Ukraine in June 2000 and further during e-mail negotiations, with Ukrainian faculty members from schools and Polish partner.
• Assessment by official organs (pending). National curriculum (OPP) for Medicine that is being constructed with the help of software is to be submitted for approval to Ministry of Education. Satisfaction of Ministry of Education requirements will check for formal presentation of data that is stored in software.

Results and products.

Our work in terms of identifying and studying of the software which can meet the requirements of Ukrainian State Standards for higher Education and may function as an instrument of curriculum development in Ukraine, has drawn the following conclusions and brought following results:

1. Ideology of State Standards for higher Education in Ukraine is progressive and meets world tendencies. At the same time the details and technological aspects of how to construct components of the Standards are not worked out properly. To overcome this problem, project team executed development of the own technology of curriculum construction.

2. International search for curriculum development instruments revealed very few choices that are very country- and specialty- (program) specific. The most attractive software that seems to fit Ukrainian needs (software based on curriculum mapping concept) is not commercially available. Therefore, project team arranged for creation of the own software product for designing national curriculum (OPP).

3. Attempts to put international standards (Canada and United Kingdom) into our software were not successful. This is explained by significant difference in formal presentation of standards between different countries. It may be concluded that software product needs additional elaboration if it is necessary to use it internationally.

4. Basing upon conclusion of non-medical experts from Research Center for Higher Education (MOE), the software may be perfectly used as a shell for any professional area as long as professional standards are being constructed within Ukrainian technology.

5. Software needs further development.

is finished and approved by the Ministry of Education, by Ministry of Health and by Scientific Committee on Medicine. Professional description (OKH) was done with some help of first module of the software and edited by experts.

is in process of development. OPP is being constructed within new curriculum designing software.

Awareness of Ukrainian and international circles regarding standard development in Ukraine and project efforts in curriculum development through intensive dissemination policy.

The common purpose of disseminative and web activities within current project was to increase knowledge of target groups regarding project efforts in development of software for State Standards of Higher Education. Carrying-out of these activities in the full scale depended upon positive results of first objective - obtaining and study of software for curriculum development.

However, from the beginning the project activities widened by development of technology for generating national curriculum (OPP) and by active involvement of project staff in dissemination of general knowledge about international and Ukrainian Standards and related information. We considered such enlightening work very much important and actively pursued it during project time. In addition, representation of Ukrainian achievements internationally allowed team to receive feedback on the results.

Information about the instrument. Prepared is a draft description of the software instrument that contains underlying concept and main short instructions regarding managing the instrument (Attachment 2). This instruction is being provided together with the instrument to people involved in instrument assessment as well as to contact audience (including DonetskDMU, National Medical University, Centre for Higher Education at Ministry of Education, Ukrainian Medical and Dentistry Academy, Lviv medical college).

Wide dissemination of information about the product is not possible because an instrument is not finished

Working groups on State Standards for Higher Education. During project period several meetings with working groups and its members were undertaken. During these meetings the ideas of instrument development, its concept, international approaches were discussed; instrument itself was demonstrated. The shortened listing of meetings is following:

May 2000 - meetings with Donetsk laboratory, Odessa, Poltava, Dnipropetrovsk, Kyiv faculty member from Ukrainian medical universities (various cities)

June 2000 - international seminar with Ministry of Health, medical schools representatives, press (overall 22 participants)

July 2000 - UMSA representative of Dentistry SSHE group

July 2000 - work on standards during Summer School funded by International Renaissance foundation (overall 17 participants)

September 2000 - Medicine, Public Health and Pediatrics working groups

October 2000 - Medicine, Public Health and Pediatrics working groups

October 2000 - Nursing working group

November 2000- Medicine, Public Health and Pediatrics working groups

November - March 2001 - OPP (national curriculum) development

December 2000 - meeting with representatives of all medical and pharmacy schools (development of 3rd component of Standards)

March 2001 - workshop in OPP development at DonetskDMU

February 2001 - meeting with representatives of all medical and pharmacy schools (development of 3rd component of Standards)

February 2001 - meeting with representatives of nursing schools (development of 3rd component of Standards)

Personal presentations to Research Center for Higher Education at the Ministry of Education.

Several events had been organized in the form of structured workshops:

November 21 - 23, 2000 - workshop of nation-wide group on Medicine standard (NMU)

December 4 - 8, 2000 - workshop on assessment (3rd component of Standards)

Jan 29 - Feb 16, 2001 - series of workshops on assessment (3rd component of Standards)

March 14 - 16, 2001 - meeting of nation-wide group on Medicine standard (NMU)

March 5 - 16, 2001 - series of workshops on assessment (3rd component of Standards)

Publishing. The cumulated knowledge in the area of educational standards development and our project experience with standard development and ways to optimize this process with the help of instrument for curriculum development have been reflected in the the first chapter of the book. The publication of book titled “Diagnostics of higher education quality” has been co-funded from current grant from International Research and Exchanges Board and by International Renaissance Foundation “Vidrodzhennya”. Calculated number of pages is 250. Circulation is 1000 copies. Book is fully ready for publishing, has been ordered and undergoes breadboarding. The table of contents is presented in Attachment 3.

Presentations and publications. Efforts on curriculum and Standard development have been widely presented on Ukrainian and international arena. The full list of publications and presentations (18 items) is reflected in Attachment 4.

Also project staff took part in educational exhibition EDUCA 2001 in Kyiv, February 2001 as part of stand for National Medical University. Stand was devoted to innovative methodologies applied in National Medical University (NMU), including standards development and introduction of objective assessment. NMU stand obtained 2nd award for innovative technologies in education.

Major Problems and Significant Delays

(Please describe steps taken to address them)

The major problems that was met by the staff during project time which caused delay in project activities are following:

• absence of available standards for most professions in Ukraine
• difficulty to obtain analogous international products for curriculum development
• absence of specified technology of curriculum development within documents that regulate creation of new State Standards for Higher Education in Ukraine
• low computer literacy of specialty experts which caused difficulties in instrument testing
• shortness of project time

The above-named problems (which are not listed in any specific order) caused certain delays in project schedule. The most significant change of project activities refers to second and third objective (dissemination effort and piloting of web-based access to the instrument respectively) where project team was forced to refuse from piloting of web-access and delay dissemination of the results. However, these activities were planned as secondary to and dependant to the 1st objective - obtaining and studying of the instrument.

Steps undertaken to overcome listed problems have already been described in “Major accomplishments” sections. Here we will just name them in brief.

In order not to paralyze project by absence of relevant professional standard, project team concentrated its efforts on health area where we could use half ready and draft standards, where we have international resources and where we can organize specialty experts for work on standards.

The largest problem was caused by absence of available curriculum development product. In time of project submission we have information about Ukrainian product for Standards development, we also used own software tool for professional description (OKH). However, search for existing instruments both in Ukraine and abroad and negotiations with owners revealed no instrument that could be obtained or purchased. Therefore, our team was forced to create necessary software instrument by us. As this activity was not planned at all, this caused the largest delay. We started programming at the end of 2nd quarter and had to execute instrument studying and work with curriculum development immediately “from the wheels”.

Prior to that, one more significant problem has arisen. It concerns insufficient specification of technology for development of national curriculum (OPP) in the regulatory documents from Ministry of Education and Cabinet of Ministers. In such situation, working groups either should create their own methodology of OPP development or make absolutely formal, meaningless documents. We have collected several approaches and developed own concept that was put in the ground of our software. This work also took time from the project.

Testing of instrument was also limited by low computer literacy of faculty members. On some stages of the project, we planned to distribute instrument for beta-testing with 2 purposes: to evaluate the instrument itself and cumulate professional description (OKH) created with the help of our instrument. However, faculty members were reasonably reluctant to master new software for one-time purpose and therefore our expectations failed. During further stages we have worked with few computer-literate experts.

And the last but not the least problem was shortness of project time. The project is administered under condition of high actuality of the topic and large scale of works. The need for curriculum development instrument is high. In our case we have been forced to postpone aggressive dissemination and web-access policy until full completeness of the product. In case of longer project we might have time to ensure execution of these works to the full extent.

Work-plan/Achievement Plan Progress - Please discuss whether each or your objectives was achieved or not, and to what degree. Please be certain to describe and include reasons for any changes in dates and or activities from the IREX-approved achievement plan/work-plan submitted to IREX.

The project “A new instrument of curriculum development” which aimed at facilitating of educational reform in Ukraine had three objectives: studying of the instrument for curriculum development, dissemination of information and piloting of web-based access to the instrument.

First two objectives are achieved to very large extent and even over-executed. Over-execution was caused by adding of new types of activities to the project: programming of instrument, preparation of the book, creating the Standard components (professional description - OKH, national professional curriculum - OPP) and even visit of international expert from the Centre for Medical Education, UK.

At the same time part of activities were not possible to execute because an instrument is still under evaluation by external experts. Basing upon expert conclusions final changes in program code may be introduced. Consequently, activities that refer to wide dissemination of ready product and information about it were limited to the contact audience (mostly working groups in health-related areas). Overall delay of second objective activities was caused by unplanned necessity to program the own software instrument.

We should also comment that in time of submitting proposal we have warned that negative results of first objective activities may postpone or prevent from execution of second and third objectives. This did not happen and objectives were executed but not in full scale.

Presented is analysis of execution of project activity workplan with comments.

Lessons Learned

• Development of new Standards is a time-, efforts- and resource-consuming process. To make this process more effective it is desirable to have special manager for one or several similar areas working groups and one computer person who can optimize the work of the group.
• A note: It is also not possible to rely on officially set dates of finishing the standards and to expect standards to become available for public immediately. Such expectation in our project that based upon Decree of the Cabinet of Ministers has caused delays and restructuring of the project.
• Curriculum development instrument that has been created and investigated during the project is not universal. It is but country-specific because it strictly follows the formal requirements of Ukrainian State Standards for Higher education. As a shell this instrument may be used for any professional area (in Ukraine). If instrument is properly filled with professional description (OKH) and professional knowledge base, it becomes profession-specific.
• Due to resource-consuming nature of Standards development, it may be reasonable to disseminate among schools the software instrument with filled-in database of professional description (OKH) - professional knowledge base - teaching blocks and established links between them. In this case the number of benefiting structures will decrease to one professional area but the potential for benefit from the software will dramatically increase because schools will have possibility for valid management of teaching blocks and linked skills and knowledge within school curriculum.
• Curriculum development Instrument is an accessory only. There still remains a lot of manual work which cannot and should not be automated. Expectations like “Software instrument will make all work for me” should be actively discouraged.
• It is not reasonable to create a software for one-time purpose (like it became with professional description (OKH) generating instrument). On the first glance there are a lot of working groups that may benefit from such instrument. However, it means that new software should be mastered by each group but once for one-time task because each group contains different specialty expert. Reluctance of professors to master new computer knowledge only reinforces this learned lesson.
• People are not eager to test software where they do not see the evident benefit for themselves. Requests and demands do not help. Other schemes for testing of software should be developed.
• It is unlikely that official governmental documents provide good description of technologies and methodologies. Other sources of information should be used.
• Project planning should allow for larger flexibility in timing and possibility to formulate activities basing upon results of prior steps.

Progress and Plans for Sustainability

(Future of partnership relationship, what training or other activities are continuing that were started under the program)

At the beginning of the project, Testing Board was interested in understanding of the role of Medical Licensing Examinations within new State Standards for Higher Education. During project period, Testing Board has undertaken the leading role in development of all components of State Standards for all health-related area. Special person (I.Filonchuk) has been hired. His sole task became management of Standard development process. Testing Board won a reputation of highly professional educational and methodological center that has a lot of relevant international and local information. Testing Board will be interested to maintain that reputation. Additionally, Ministry of Public Health of Ukraine has a close eye on the process of Standards development.

All that means that the work on development and completing of Standards will continue till the winning end. This work will be done in cooperation with working group (which has to work according to regulatory documents) and supported by at least 3-4 medical schools that take active part in medical educational initiatives.

It also means that Testing Board has direct interest in the software instrument. This interest and capacity of the Testing Board and will ensure it maintenance and dissemination even after the project.

Comments on partnership - IREX relations, suggestions for IREX.

Our collaboration with International Research and Exchanges Board (IREX) during this project left the very positive impressions. It may be confirmed by several following facts:

• At the beginning, project manager and project accountant obtained sufficient instructions in terms of requirements to project execution, documentation ad reporting. During this instructing we have been provided with all necessary financial forms which significantly eased our financial work on the project and outlines of quarter and final reports.
• Requirements for financial and programmatic reporting are not overwhelmed and well structured that help in report preparation. Therefore, reporting takes reasonable amount of time that is important considering low budget and short period of the project.
• There have been several site visits during project time that left impression about IREX officers as a professional, knowledgeable and intelligent staff.
• We also feel flexibility of IREX and openness to need of their grantees.

THANK YOU.

which were negotiated for use in curriculum development

1. “Standard-based education in Ukraine: are we going I the right direction?” I.Bulakh, Y.Voronenko, Y.Bogachkov. in AMEE Conference in Beer-Sheva “Horizon scanning in Medical Education: 2020 Vision” August 2000.

2. “Supporting the introduction of objective national testing” M.Mrouga, L.Artemchuk in AMEE Conference in Beer-Sheva “Horizon scanning in Medical Education: 2020 Vision” August 2000.

3. “Standard-based education in Ukraine: are we going I the right direction?” I.Bulakh, Y.Voronenko, Y.Bogachkov.//Medical Teacher, #2, 2001.

4. “Supporting the introduction of objective national testing” M.Mrouga, L.Artemchuk .//Medical Teacher, #2, 2001.

5. “Information support of standardizing of higher education” Y.bogachkov, V.Petrenko, N.Moklyak in “Automatics 2000”, Lviv, September 2000.

6. “Place of testing in standardizing of higher education” V.Petrenko, Y.Bogachkov. //Digest of scientific articles of Institute of problems of modeling in energetic sector. Issue 9, NAN of Ukraine, Kyiv, 2000.

7. “Model of development of educational and professional program basing upon educational and professional description (OKH)” Y.Bogachkov, M.Mrouga// Digest of scientific articles of Institute of problems of modeling in energetic sector. Issue 2, NAN of Ukraine, Kyiv, 2001.

8. Educational exhibition EDUCA 2001. Kyiv, February 2001.

9. “About new Standards for Higher Medical Education of Ukraine” I.Bulakh. Presentation at meeting of Rectors of Medical Schools. Vinnitsa, March 2001.

10. “Testing as a method of standardized assessment in modern higher educational establishment” I.Filonchuk. Presentation at seminar in Interregional Academy for Personnel Management (MAUP). March 2001.

11. Seminar “Role of Standardized Testing in Curriculum Development“. Approved by IREX Alumni. June 2001.

12. “A new instrument of curriculum development: curriculum as a function of professional outcomes” I.Bulakh, Y.Voronenko, I.Filonchuk. Accepted for AMEE Conference in Berlin. August 2001.

13. “The dynamics of knowledge structure of graduating medical students based on results of medical licensing examination” M.Mrouga, L.Artemchuk, Y.Bogachkov. Accepted for AMEE Conference in Berlin. August 2001.

14. “Development of new Standards for Higher Medical Education” I.Bulakh, I.Filonchuk. Accepted for research and educational conference at Dnipropetrovsk State Medical Academy. October 2001.

15. “Analysis of advantages and problems of national medical licensing examinations” M.Mrouga, I.Bulakh. Accepted for research and educational conference at Dnipropetrovsk State Medical Academy. October 2001.

16. “Standards in Health Education” I.Bulakh, I.Filonchuk. Accepted for educational conference at Kyiv Medical Academy of Postgraduate Education. November 2001.

17. “Results of national medical licensing examinations in 2000/2001 academic year” M.Mrouga, L.Artemchuk. Accepted for educational conference at Kyiv Medical Academy of Postgraduate Education. November 2001.

18. “Diagnostics of higher education quality”, book, in print.

Annex 5.

List of products (course materials, videos, handbooks, etc.)

developed if possible. Please include title and brief description of material.

1. Book “Basic principles of educational measurement”.
The book is a primer on educational innovations, mainly devoted to standards and objective assessment. The first chapter is devoted to different aspects of standard development. Book bases upon materials of summer school funded by International Renaissance Foundation “Vidrodzhennya” and amended with materials from studying of curriculum development instrument. Calculated number of pages - 250. Planned circulation - 1000 copies.

2. Software instrument for curriculum development.

This is an database which contains forms for professional description (OKH), professional knowledge base and teaching blocks with tools for establishing links and viewing of data by different grouping and criteria.

3. Professional description of Medicine (OKH).

First part of State Standards for Higher Medical Education which was created with partial help of software instrument and has already been approved by relevant organization: Ministry of Education, Ministry of Health, Commission for Medicine. Professional description (OKH) presents the full list of skills that should be possessed by medical doctors after graduation.

4. Medicine-specific instrument for curriculum development.

The shell filled with linked educational data: professional description for Medicine (OKH), professional knowledge base extracted from test outline for Medical Licensing Examinations and teaching disciplines which currently exist in medical schools of Ukraine.

5. National professional curriculum for Medicine (OPP)

- in process of development.