The tests are predicated on the idea that mere technical ability is not sufficient for information literacy, rather it is composed of seven dimensions that a person should be able to use ICT in: “Define”, “Access”, “Manage”, “Integrate”, “Create”, “Communicate” and “Evaluate”.
In order to assess the pupils’ abilities in these seven dimensions, CITE defined for each teaching subject tested—Chinese Language, Mathematics, and Science (Biology)— a set of tasks where each segment tested one of these dimensions.
In addition, questionnaires were distributed to:
the heads of the schools involved, to see what they prioritised in terms of pedagogical and technical goals at their schools,
the teachers at the schools, to get their own assessments of how well they understood ICT and how to use it in teaching,
the ICT staff at the schools, to get their assessment of what ICT software and hardware were available at the schools and what teaching activities they could support,
the tested pupils, to see if they had access to computers outside of school, how much they used them and for what.
The test tasks were set up on servers at CITE and accessed by software clients installed on the computers of the participating schools.
The tasks would require the pupils to:
Define: What information do they need to solve the task? I e, how should they search for information, and what keywords should they use?
Access: Retrieve information from online sources.
Manage: Store and arrange data on their computers.
Integrate: Combine and analyse the retrieved data.
Create: Create a document presenting the integrated data.
Communicate: Present the integrated data to a specific audience.
Evaluate: Determine the extent to which the collected data fulfil the requirements.
The exact data to be collected and how it would be presented was chosen for each school subject under test.
 Law, N., Yuen, A., Shum, M. and Lee, Y. 2007. Final Report on Phase (II) Study on Evaluating the Effectiveness of the ‘Empowering Learning and Teaching with Information Technology’ Strategy (2004/2007). Centre for Information Technology in Education (CITE), Faculty of Education, The University of Hong Kong.
The major weakness is that the process is hugely labour-intensive: It requires setting up servers and developing software for running the tests, installing client software at all the participating schools, running tests with invigilators in the room, and then evaluating the answers afterwards. (In actuality, not every response was in fact evaluated, but a representative sample, a full evaluation being impossible.) Due to the element of a creative process, there is no simple, automated way of evaluating the responses.
The evaluators were trained to grade the responses in a uniform way, and double-checking their reported grades confirms that they managed to do this.
There are other limitations, in that the time allotted to the tests seemingly was too short, but knowing this, it is of course possible to extend the time, should one replicate the test, but this would on the other hand introduce further resource requirements.
Conclusions from the study show that in general students at all levels attained the basic level in all dimensions but were “rather weak” in attaining higher levels of proficiencies that required higher-order or critical thinking. The study found the strategy to be generally effective but there were still gaps and discrepancies among schools in terms of support and infrastructure. Teachers were found to be more competent in general ICT use than the pedagogical use of ICT.
The recommendations from this study were that the HKSAR Government should establish a minimum standard in terms of ICT access and school’s ICT infrastructure. To add to this they noted that although guidelines for the employment of technical support staff exist, there is no enforcement mechanism to ensure that such guidelines are being appropriately used by schools and recommended to be established is an up-to-date benchmark test for the minimum expected knowledge and skills of school technicians.