Caja PDF

Comparta fácilmente sus documentos PDF con sus contactos, la web y las redes sociales.

Compartir un archivo PDF Gestor de archivos Caja de instrumento Buscar PDF Ayuda Contáctenos



WPS6851 .pdf



Nombre del archivo original: WPS6851.pdf
Título: World Bank Document
Autor: World Bank Group

Este documento en formato PDF 1.4 fue generado por Acrobat PDFMaker 11 for Word / Adobe PDF Library 11.0 / PDF PT 3.40p (pdf-tools.com), y fue enviado en caja-pdf.es el 29/06/2014 a las 10:25, desde la dirección IP 95.61.x.x. La página de descarga de documentos ha sido vista 1526 veces.
Tamaño del archivo: 1.8 MB (34 páginas).
Privacidad: archivo público




Descargar el documento PDF









Vista previa del documento


Public Disclosure Authorized

Policy Research Working Paper

Public Disclosure Authorized

6851

Which World Bank Reports Are Widely Read?
Doerte Doemeland
James Trevino

Public Disclosure Authorized

Public Disclosure Authorized

WPS6851

The World Bank
Development Economics Vice Presidency
Operations and Strategy Unit
May 2014

Policy Research Working Paper 6851

Abstract
Knowledge is central to development. The World Bank
invests about one-quarter of its budget for country
services in knowledge products. Still, there is little
research about the demand for these knowledge products
and how internal knowledge flows affect their demand.
About 49 percent of the World Bank’s policy reports,
which are published Economic and Sector Work or
Technical Assistance reports, have the stated objective
of informing the public debate or influencing the
development community. This study uses information on
downloads and citations to assesses whether policy reports

meet this objective. About 13 percent of policy reports
were downloaded at least 250 times while more than 31
percent of policy reports are never downloaded. Almost
87 percent of policy reports were never cited. More
expensive, complex, multi-sector, core diagnostics reports
on middle-income countries with larger populations tend
to be downloaded more frequently. Multi-sector reports
also tend to be cited more frequently. Internal knowledge
sharing matters as cross support provided by the World
Bank’s Research Department consistently increases
downloads and citations.

This paper is a product of the Operations and Strategy Unit, Development Economics Vice Presidency. It is part of a larger
effort by the World Bank to provide open access to its research and make a contribution to development policy discussions
around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors
may be contacted at ddoemeland@worldbank.org.

The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development
issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the
names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those
of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and
its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

Produced by the Research Support Team

Which World Bank Reports Are Widely Read?1
Doerte Doemeland and James Trevino

JEL Codes: O19, H87, and D73.
Keywords: Knowledge Management, International Organizations, Internet.

1

The authors would like to thank Zeljko Bogetic, Simon Davies, Jean-Jacques Dethier, Kene Ezemenari, Sharokh
Fardoust, Aart Kraay, Eileen Monnin-Kirby, Jimmy Olazo, David Rosenblatt, Zia Qureshi, Adam Wagstaff and
participants at a DEC seminar in December 2013 for excellent comments and suggestions, and Eliza McLeod,
Yoshimi Muto, Irina Sergeyeva, and Jeannette Smith for assistance with the data.

ACRONYMS
ESW

Economic and Sector Work

LEG

Legal

AAA

Analytical and Advisory Activities

MNA

Middle East and North Africa

AFR

Africa

OCR

Optical character recognition

D&R

Documents & Reports

OKR

Open Knowledge Repository

DEC

Development Economics

OMBC Online Media Briefing Center

DL

Downloads

OPCS

Operation Policy and Country Services

DO

Development Objectives

PAR

Performance assessment review

EAP

East Asia and Pacific

PPP

Purchasing Power Parity

ECA

Europe and Central Asia

PREM

Poverty Reduction and Economic
Management

FCO

Fund Center Owner

QAG

Quality Assurance Group

FPD

Financial and Private Sector Development

RePEc

Research Papers in Economics

FY

Fiscal Year

SAR

South Asia

GDP

Gross Domestic Product

SDN

Sustainable Development Network

HDN

Human Development

TA

Technical Assistance

IBRD

International Bank for Reconstruction and

TTL

Team Task Leader

Development
IDU

Internal Documents Unit

VPU

Vice-presidential Unit

IEG

Independent Evaluation Group

WBI

World Bank Institute

LCR

Latin America and Caribbean

WDI

World Development Indicators

2

I. Introduction
Knowledge is central to development. It is instrumental for developing new products. Knowledge can
also help produce existing products more efficiently, generate better outcomes, and enable policy
makers to make better choices. Knowledge absorption has been found to drive productivity and income
growth and to contribute significantly to poverty reduction.
Large international organizations, such as the World Bank, can play an important role in generating
and transferring development knowledge that is relevant for economic development. They can draw
knowledge from cross-country experiences to improve diagnostics and provide better policy advice.
They can also make knowledge available as a public good that can be utilized in many sectors of
development activity and in many countries. The production of knowledge goods is often not costefficient at the country level and many developing countries do not have the capacity to develop these
types of goods.
The World Bank spent about one-quarter of its country services budget on core knowledge products
in fiscal year 2012. Core knowledge services 2 are: i) economic and sector work (ESW), ii) technical
assistance (TA), iii) the World Development Report iv) external training and capacity development, v)
research, vi) impact evaluations, vii) global monitoring, viii) new product development, and ix) internal
reports. 3 These knowledge products can be divided further into knowledge for external audience and
knowledge for internal use. Spending on the World Bank’s “core” knowledge tasks has increased steadily
over the past decade. Expenses on knowledge for external clients and public goods accounted for
approximately 83 percent of expenses on core knowledge services in FY2012, of which 74 percent were
costs for knowledge products for external clients and 9 percent associated with the development of
public goods. ESW is the most important knowledge product for external clients next to TA, absorbing
roughly 22 percent of total costs of the nine core knowledge products (World Bank, 2012a). ESW and TA
are also the only two core knowledge products that form part of country services.
Internal knowledge sharing is essential for a large and complex institution such as the Bank to provide
effective policy advice. Bottlenecks to information flows create inefficiencies, either through duplication
of efforts and diverting resources from knowledge creation itself (World Bank, 2000). The idea behind
establishing the Bank’s networks in 1996 was to ensure a flow of knowledge throughout the Bank.
Internal knowledge sharing can take place through several channels, such as cross support, training, and
internal knowledge products. The latter includes the dissemination of sector reports and policy papers
via seminars, the creation of tools and databases, and training for Bank staff. But internal knowledge
transfer is not only provided through these targeted products. Improving internal knowledge sharing has
gained renewed importance in the context of current WB Change Process. The ongoing reforms aim for
the World Bank to become a catalyst for global knowledge by connecting practitioners and by
supporting networks of researchers, policy makers, and civic organizations keen to learn about what
works and how to implement successful results. The knowledge acquired is used to diffuse innovation at
scale, so that successful projects and programs are replicated under the right conditions or with the
right adjustments by practitioners across the world (World Bank, 2013a).

2

Non-core knowledge products absorbed about US$300 million in fiscal year 2010. They include products such as
country partnership strategies, sector strategies and evaluations by the Independent Evaluation Group.
3
See World Bank (2011a) for further elaboration.

3

An important channel through which internal knowledge transfer occurs in the World Bank is cross
support. Cross support is generally defined as “staff time of an expert or specialist purchased from
outside the responsible unit for a specific task” (IEG, 2012, pg. 56).4 Tasks can be both operational and
knowledge-generating in nature. Cross support can take the form of participation in team visits,
preparation of key inputs for analytical and advisory activities (AAA), and peer review. Cross support is
short-term by nature and does not include staff movement or rotation.
There is little systematic research about the demand for and use of World Bank knowledge products.
Wagstaff (2012a) assessed the distribution of downloads among World Bank published ESW reports and
finds an average of 123 downloads for the 799 reports in his sample. He argues that web-based metrics,
such as downloads, could provide a useful tool for assessing demand for World Bank knowledge
products. He does not identify the factors that determine the number of downloads. Ravallion and
Wagstaff (2010) utilized Google Scholar to generate citation counts for a large quantity of books and
research publications of the Bank. 5 They found that the Bank’s research record in development
economics was on par with leading economics departments, but that a large portion of the Bank’s
research goes uncited. They also note that citations are dependent on the purpose of the article and
the intended audience. Factiva, a research tool that catalogues over 28,500 media sources, was used by
Reeves (2011) to measure the coverage of World Bank publications. She found that 1,442 out of 2,085
Bank titles received media coverage within the three year period after their publication.
We find that more expensive policy reports on populous middle-income countries are more likely to
be downloaded and cited, especially if these reports have the stated objective of informing the public
debate. 6 We find that more complex policy reports, such as those that focus on multiple sectors or core
diagnostic reports, are also more likely to be downloaded and cited. During the past 5 years the Bank
finalized an average of 322 policy reports per year, of which 49 percent have the stated objective of
informing the public debate. About 13 percent of all policy reports are downloaded at least 250 times,
while about 32 percent are never downloaded. Over 31 percent of policy reports are never downloaded,
while about 13 percent are downloaded at least 250 times. Almost 87 percent of policy reports were
never cited but multi-sector reports tend to be cited more frequently. Internal knowledge sharing
matters as cross support provided by the World Bank’s Research Department (DEC) consistently
increases downloads and citations.
The objective of this study is two-fold: first, we provide objective estimates of the demand for and use
of the World Bank’s policy reports; second, we discuss the roles that different costs play with regards to
the demand and use of policy reports. Since the generation and transfer of development knowledge by
the World Bank is important in facilitating its objectives, getting a better understanding of the quality
and impact of the Bank’s knowledge work is paramount.
This study is structured as follows: Section II describes the data on policy reports. Section III discusses
the measures of demand for policy reports used in this study. Section IV presents data and
4

Cross support is generally defined as support of staff across vice-presidential units (VPUs). Cross support within
VPUs is even more frequent and is key for the World Bank to deliver multi-sector knowledge productions and
operations.
5
It included articles, working papers, books, edited volumes, chapters, and conference proceedings.
6
For the purpose of this study, policy reports are defined as those ESW and TA reports that have been published.

4

methodology. Section V summarizes results. Section VI discusses the implication of World Bank internal
knowledge sharing. The final section concludes.

II. Generating Knowledge in the World Bank: External Policy Reports
The World Bank has three key types of core knowledge products: external client knowledge, public
goods, and knowledge for internal use. Over three-quarters of expenditures on knowledge products are
for external clients (Figure 1a). There are four distinct external knowledge products: Economic and
Sector Work (ESW), Technical Assistance (TA), Impact Evaluation and external training. Along with TA,
ESW has increasingly become a core part of the Bank’s engagement with clients—in fiscal years 2000–
2006 the Bank spent 26 percent of its spending on country services on ESW and TA (IEG, 2008). Between
April 2012 and March 2013, expenditures for these two products constituted 39 percent of its spending
on country services, a substantial increase from previous years (World Bank, 2013b, Figure 1b).
Figure 1: Distribution of Expenditures
a) Distribution of Expenditures of
Knowledge Products

1%
0%

15%

b) Distribution of Expenditures on Country
Services

8%

External Clients

Project Supervision

6%
31%

8%
Public Good
Internal Use

Lending Preparation
Economic and Sector Work
Technical Assistance

27%

Aid Coordination
14%

77%
13%

Country Program Support
Client Training
Impact Evaluation

Source: World Bank Quarterly Business and Risk Review FY13 Quarter 3.
Note: Data is for the period Q4 FY12 to Q3 FY13.

Economic and Sector Work is the World Bank’s primary knowledge product line and often
complements technical assistance (TA). It is the World Bank’s primary country-based analytical and
advisory business line, intended to provide a basis for i) conducting policy dialogue; ii) developing and
implementing country strategies; iii) formulating effective lending programs; iv) building institutional
capacity and informing the international community about a country’s development challenges. It must
involve original analytical effort, be undertaken with the intent of influencing an external client’s policies
and program, and be owned by a specific Bank unit (World Bank, 2012b). 7 The product represents the
view of the Bank, and as such is not attributable to individual authors. ESW is distinct from public good

7

There are three ESW report types: First, core diagnostic reports, which include Country Economic Memoranda,
Development Policy Reviews, Poverty Assessments and Public Expenditure Reviews; second, other diagnostic
reports, which cover a range of sector-specific topics; and third, advisory reports, which address high priority
sector-specific/thematic issues. Each type of diagnostic report has its own guidelines (World Bank, 2002).

5

research8 in that it is a knowledge product meant to address specific questions of client countries. This
client-demand-driven approach aims to support the development of country strategies and helps with
the formulation and implementation of lending programs. While ESW involves research and analysis
and is meant to inform policy choices, TA is mostly advisory. Nonetheless, technical advice on
formulating and implementing policies and programs can result in report outputs; in most cases,
however, it takes the form of the provision of TA, on demand advisory services, or training.
For the purpose of our study we rely on the policy reports within the D&R database. Documents &
Reports is a database that contains more than 130,000 publically available World Bank documents. We
define policy reports as those documents within the D&R database that were filed as ESW. These are
either ESW or published TA reports. There have been, on average, 322 policy reports per year during the
past five years of which around 250 were ESWs. This implies that approximately 52 percent of ESW
projects do not have a corresponding report in the Documents & Reports (D&R) database. This is due to
three factors. First, not all ESW are designed to produce reports. Second, some reports are confidential.
Though the World Bank’s current Access to Information policy presumes that ESW report will be
immediately disclosed, some reports can be flagged as confidential (World Bank, 2010a; World Bank,
2013c).9 Third, some ESWs that have produced reports may not have been filed with the World Bank’s
Internal Documents Unit (IDU).
Many policy reports are part of a larger series with several budget codes. In order to identify key
characteristics of policy reports, such as producing unit, cross support received or costs, we merge
information from the D&R database with the budget codes associated with the different reports.
However, not all policy reports have a unique budget code. There are three distinct cases: First, multiple
reports were funded under a single policy report. 10 In many cases there are only two or three reports
under a single budget code, but in a few cases there are as many as 12 reports under a single budget
code. Second, there are single reports that were funded under multiple budget codes. 11 Overall, there
were 149 distinct codes that were aggregated into 66 policy reports. Third, there were multiple budget
codes that were linked to multiple reports. Some of the groupings reflected large thematic report series.
Of the original 1,765 codes and the 2,020 reports, we are left with a dataset of 1,611 policy reports of
which 1,331 are documents that were released under a single report number that was funded under a
single code.

8

Public good research of the World Bank includes Open Data, Policy Research Working Papers, World
Development Report, journal articles, and books, among other things.
9
Over the past 15 years, the World Bank’s policy on disclosing information has evolved gradually. Prior to 2010,
the World Bank’s approach had been to spell out what documents the World Bank discloses. Under the World
Bank Policy on Access to Information which became effective on July 1, 2010, the World Bank discloses any
information in its possession that is not on a list of exceptions. Documents flagged as “official use only” are
disclosed 5 years after preparation, while those flagged “confidential” or “strictly confidential” are disclosed after
20 years. Confidential reports, for example, include reports with information that has been provided by member
countries in confidence or analysis that may affect financial marked behavior.
10
These were sometimes simply a single document that was translated into other languages and released under
separate budget codes. For these reports, we created one aggregate report identifier since we could not distribute
the expenditures linked to a project code across the multiple report numbers.
11
In April 2012, a Programmatic Approach for ESW and TA was introduced to organize AAA of multiple program
activities and knowledge products that support a particular program, theme, or engagement area over several
fiscal years (World Bank, 2012b). Since our data set is limited to fiscal year 2012, it does not include any reports
categorized as programmatic AAA.

6

The majority of policy reports are produced by regional vice presidential units (VPUs). The Africa
region (AFR), East Asia and Pacific region (EAP), and Europe and Central Asia region (ECA) generated
more than 52 percent of knowledge products between fiscal years 2008–2012 (Figure 2). For the
networks, Finance and Private Sector Development (FPD) and Sustainable Development (SDN) have
historically taken up the largest share of policy reports, but in the past few years Poverty Reduction and
Economic Management (PREM) has increased its number of policy reports, albeit from a very low level.
The Middle East and North Africa (MNA) produced the smallest number of reports among the regions,
while the Human Development (HDN) Anchor produced the least among the networks.
Figure 2: Policy Reports per Year
Policy Reports per Year

368

100

0

0

300

275

SDN
PRM
HDN
FPD
SAR
MNA
LCR
ECA
EAP
AFR
WBI

200

303

100

Number of Policy Reports

337

200

300

328

Number of Policy Reports

400

400

Policy Reports per Year

2008

2009

2010

2011

2012

2008

Source: World Bank Documents & Reports and World Bank SAP database.

2009

2010

2011

2012

Source: World Bank Documents & Reports and World Bank SAP database.

III. Measuring Transfer of Knowledge
Individual knowledge services can be evaluated against their specific objective. For instance, if
knowledge services are meant to improve the design of lending operations, measures can be used to
assess whether the quality of operations has improved. If they are intended to improve decisions by
customers or stakeholders, measures can be used to assess whether these partners believe their
decisions were positively shaped by the knowledge they received. If knowledge services are aimed at
generating public goods, it can be assess whether and to what extent those public goods are having the
intended outcomes.
All policy reports are required to have a development objective. Task team leaders (TTL) need to
specify a clearly defined development objective, as well as intermediate outcomes of the report and
risks to achieving the desired results when creating a budget code for an ESW or TA in the system. They
can choose among one or more of five predefined development objective categories for an ESW or TA: i)
informs bank lending); ii) informs government policy; iii) builds capacity; iv) informs the development
community and v) stimulates public debate. Figure 3 below shows the distribution of development
objectives among our policy report dataset. The development objective of about 55 percent of policy
reports is to inform government policy and of 49 percent to inform public debate. Correlation among
the different development objectives is weak. The three pair-wise correlations between building client

7

analytical capacity, informing the public debate, and influencing the development community are in the
range of 0.23 and 0.26. 12
Figure 3: Objectives of Policy Reports
55%

60%

49%

50%
40%
30%

31%

31%

32%

Build client
analytical
capacity

Inform/stimulate
Influence
No Development
public debate
development
Objective
community

30%

20%
10%
0%
Inform Lending

Inform
government
policy

Source: Business Warehouse.
Note: Policy reports can have multiple objectives.

Increased focus on the quality of knowledge services and its impact has led to a three-pronged
approach towards measuring quality: i) self-assessment, ii) systematic collection of client feedback, and
iii) IEG performance reviews.
i) Self-assessments are provided by the TTL of the report. After completing a policy report, the TTL is
required to evaluate in the system whether the report has met its pre-specified objectives. 13 The Sector
Manager and Country Director later validate and endorse this assessment. Table 1 below shows that
TTLs assess their work favorably on average. Almost half of all policy reports that had the objective of
informing lending were considered to have fully met this objective, 14 while more than a quarter of policy
reports that sought to influence the development community had the highest assessment score.
Informing and stimulating the public debate was the highest rated objective. About 46 percent of policy
reports with this objective had fully met it according to the self-assessment.

12

Recent Independent Evaluation Group (IEG) studies show that only 23 percent of knowledge services included
indicators that tracked the achievement of the policy reports outcome, and that those knowledge services that had
tracking mechanisms performed better in terms of meeting the specified objectives (World Bank, 2013d).
13
The development objectives mentioned earlier are measured using a scale ranging from 1 to 0. A score of 1
indicates that the objective was fully met; a score of 0.75 or above indicates that it was largely met; a score of 0.5
or above indicates that it was partially met; and a score of 0 indicates that the objective was clearly not met.
14
Many are ESWs are undertaken with the objective of providing the analytical basis for future lending operations.
This indicator is such likely to capture management foresight and political risks (for example, a planned lending
operation may go ahead because of a change in government) rather than the quality of the ESW.

8

Table 1: Policy report Average Self-Assessment Scores

Year
2008
2009
2010
2011
2012
2008–2012
Reports with DO
% of Policy reports
with a score of 1

Inform
Lending
0.79
0.79
0.79
0.75
0.85
0.79
484
49%

Development Objectives (DO)
Inform
Build Client
Government
Analytical
Inform/Stimulate
Policy
Capacity
Public Debate
0.72
0.76
0.81
0.72
0.71
0.84
0.72
0.74
0.83
0.72
0.77
0.84
0.72
0.71
0.83
0.72
0.74
0.83
861
489
773
32%

31%

46%

Influence
Development
Community
0.69
0.67
0.71
0.70
0.75
0.70
496
26%

Source: Business Warehouse.
Note: A score of zero indicates that the DO was not met; a score of one indicates that the DO was fully met.

ii) Client feedback on knowledge products is routinely being sought. Recently, the World Bank has
begun to systematically gather and incorporate client feedback for ESW and TA. The Bank sends surveys
to government counterparts of selected ESWs and TA to seek their view regarding the quality,
relevance, and impact of the provided knowledge service. The most recent survey was completed at the
beginning of FY13. Furthermore, work is ongoing to have client feedback information disseminated for
economic sector work, non-lending technical assistance, external training and internal knowledge
products (World Bank, 2012c). A product-specific survey was sent to users of ESW and TA for 210
projects that were completed in FY12, with preliminary results showing that many of these products
were considered by their users to be effective at achieving their agreed objectives. The effectiveness of
these reports was measured on five characteristics: i) how effective they were at addressing the specific
development goals of their agency, ii) their technical quality; iii) their use of best available data, iv) how
effective they were in engaging the clients during the design, implementation, and completion of the
work, and v) whether the product or service was delivered in a timely manner. Table 2 below shows the
descriptive statistics of this most recent report, measured on a six point scale, with 1 being considered
to be “Very Ineffective” and 6 considered to be “Very Effective.” As mentioned in World Bank (2013b),
the average ratings for quality and likely impact are deemed to be effective, with median ratings of 5 on
a 6 point scale. This is commensurate with the self-assessment that TTLs gave regarding whether their
policy report informed public debate. The client survey also asks whether or not the knowledge product
or activity has led to specific changes, either through policy, regulations, or institutional changes.
According to the latest survey, reports scored very high with respect to relevance and technical quality
and more than three-quarter led to a change in policies, regulations or institutions. 15

15

There are a few shortcomings with the client feedback surveys. First, the survey is likely to suffer from significant selection
biases arising from a high non-response rate and the fact that the TTL could choose the respondents. Feedback was provided
only for 113 out of 210 projects for the question over whether the policy report had led to a policy change within the client
agency or institution and the number of feedback providers varied significantly across reports. Second, feedback was requested
for a smaller number of projects than in our dataset (210 ESW and TA projects were analyzed in the client feedback survey for
FY10, while our dataset covers 275 policy reports).

9

Table 2: Client Feedback Indicators for FY 2012
Effectiveness in terms of: Count Mean
Standard Deviation
Achievement of agreed objectives 192
4.89
0.79
Relevance 195
5.06
0.75
Technical Quality 191
5.05
0.79
Engagement 195
4.95
0.81
Timeliness 192
4.60
1.00
Policy Change? (Yes=1) 113
0.77
0.38

Median
5.00
5.00
5.00
5.00
5.00
1.00

Source: World Bank Operation Policy and Country Services.
Note: The Effectiveness indicators are measured on a scale of 1 to 6, while the Policy Change indicator
is a binary indicator.

iii) Performance assessment reviews are also used to evaluate policy reports. Performance assessment
reviews (PAR) are part of the IEG initiative to assess the impact of Bank ESW and TA. This initiative is
meant as a replacement for the evaluations formerly performed by the Quality Assurance Group (QAG),
which was disbanded in FY10. Until now there has not been an autonomous review process of AAA (IEG,
2012). These ratings consist of the following categories: results, relevance, technical quality, and
dialogue/dissemination. All measures use six-point scale ranging from “highly satisfactory” to “highly
unsatisfactory”. They are based on four sources: reviews of the content of the policy report to establish
substantive content; reviews of the documentary record to probe into process issues, including
inception, client and stakeholder engagement, quality control, and dissemination; interviews of country
directors and task managers that were familiar with the reports as well as the network responsible for
the reports; and interviews of government officials and other stakeholders in the public and private
sectors in the client countries as well as staff in the resident offices (World Bank, 2010b; World Bank,
2011b). 16
This study measures the demand for and use of policy reports through downloads and citation counts.
These two indicators are objective measures which serve as a complement to the three abovementioned approaches to measuring the quality of these types of knowledge products.
Download counts capture the intent to use World Bank policy reports. More specifically, download
counts refer to the number of times a PDF has been downloaded from the World Bank’s external
website. As noted in Wagstaff (2012b), downloads are an excellent indicator of the use of knowledge
created by the Bank, particularly the intent to use the document since it is reasonable to assume that
the person who downloads a policy report would at least take a look at the contents of the document.
We consider this indicator an objective measure of whether policy report meets the development
objective of informing the public debate.
Citations counts measure how often a report was cited by other publications. In academia, research
papers cite other publications that they have used in performing research. Citations are considered a
good indicator of the influence of academic research and are widely used for this purpose in all fields.
While citations are a commonly used metric for analyzing the impact of published academic articles,
they have not been used to assess the demand of policy reports, in part because policy reports are

16

While these documents are available within IDU, there does not appear to be an organized database that
contains these reviews, nor is there an official guideline provided by IEG explaining the methodology of PARs.

10

written for policy makers rather than researchers. Still, World Bank policy reports are often cited by
think-tanks, publications from other donors or government institutions. Using a broad search engine,
that is not restricted to a research network, we argue that citations can used an alternative objective
measure to assess the influence of a policy report on the development community.

IV. Data and Estimation
Data of downloads was gathered for all policy reports which are part of the World Bank’s Documents
and Records (D&R) database. The D&R database contains well over 130,000 documents, but as noted in
Wagstaff (2012a) it does not include all documents produced by the Bank. Still, it is the most complete
data base of published World Bank documents. D&R also provides a URL link to the actual document, in
PDF form, which proved critical in facilitating the collection of download information. One potential
issue regarding the collection of download data is the possibility that some policy reports could
additionally be hosted on databases other than D&R. Policy reports could be downloaded from other
websites such as the Research Papers in Economics (RePEc) which would not be captured in our data,
but this is unlikely to significantly affect any of the results presented below.
Download counts were gathered using Omniture web analytics software. First, we created a script to
scrape the document web addresses for all policy reports from the D&R website to be able to identify
documents not only by title or report number but also web address. We then matched the data from
D&R (which had relevant information on the Project Code, Report Number, and document language,
and title) with the information on all downloads obtained with Omniture. We were able to identify from
Omniture how often policy reports were downloaded, and when they were downloaded. 17
Policy reports in English receive the largest number of downloads. Figure 4 shows the total number of
downloads by VPU broken down by the language of the policy report. Despite there being six working
languages within the Bank (English, French, Spanish, Russian, Arabic, and Chinese), 74 percent of the
policy reports in our dataset are published in English. Still, a significant number of reports are published
in languages spoken within the region, such as Spanish language reports in the Latin American and
Caribbean region (LCR), French language reports in AFR and MNA, Arabic language reports for MNA,
Chinese language reports in EAP, and Russian language reports in ECA. The largest number of policy
reports in other languages is in EAP, attributable to the highly-downloaded reports covering Vietnam
and Indonesia. Generally speaking, policy reports owned by the East Asia and the Pacific region had the
most downloads. In fact, there are fewer policy reports for the EAP region than for the AFR region
(Figure 2), yet there are almost twice as many downloads. The low number of downloads for policy
reports owned by Networks is to some extend driven by the low number of reports produced by these
VPUs.

17

For this report we kept the granularity of the data to an annual basis, but data is also available on an hourly,
daily, and monthly basis with Omniture. We were not able to see from where reports were downloaded, as this
option is only available for webpage views within Omniture.

11

Figure 4: Policy Report Downloads by language and VPU Figure 5: Download Distribution
Downloads Distribution of Policy Reports: 2008 to 2012

Downloads from 2008 to 2012 by VPU
400
300
200

Number of Reports

0

100

40,000

Other
Chinese
Arabic
Russian
Spanish
French
English

20,000

Downloads by Language

60,000

All Reports with Downloads

0

1000

2000

0

Downloads per Report

WBI AFR EAP ECA LCR MNA SAR FPD HDN PRM SDN OTH

Source: World Bank Documents & Reports and World Bank Omniture.

Source: Omniture and World Bank SAP database.

Downloads follow a highly skewed pattern. This distribution, seen below in Figure 5, is typical for
count data: datasets that track the incidences of an action, such as downloads, are highly centered on
zero and low numbers. A large portion of policy reports were downloaded relatively few times: Almost
40 percent of policy reports were downloaded between 1 and 100 times. The “knee of the curve” 18 of
the dataset occurs around 250 downloads. Those policy reports that were downloaded more than 250
times compose 13 percent of our sample. There are only 25 policy reports (2 percent of the dataset)
that have more than 1,000 downloads during the period investigated (FY2008 to FY2012). Over 31
percent of the policy reports in our dataset (517 out of 1,611) were never downloaded. It is, however,
important to keep in mind that many policy reports were not intended to reach a large audience but
prepared to assess very specific technical questions or inform the design of lending operations.
Policy reports that have been released for a longer period of time are downloaded more often.
Average downloads per document and the number of policy reports that have been downloaded drop in
recent fiscal years. Downloads of reports decline over time: Policy reports have an average of 1.6 daily
downloads during their first year of release, which decreases to 0.6 downloads during their second year
and approximately 0.4 downloads during the third year. Twenty-five policy reports that have been
downloaded more than 1,000 times all were released between fiscal years 2008 and 2010. When these
outliers are excluded from the dataset, the average downloads are reduced to 87 for the time period.
Table 3 shows the summary statistics for the cumulative downloads for policy reports.

18

The “knee of the curve” is the point at which the derivative of the curve is transitioning from a value greater
than one to a value less than one.

12

3000

Fiscal Year
Published
2008
2009
2010
2011
2012
2008–2012

Total
Published
303
328
337
368
275
1611

Table 3: Download Statistics for policy reports
Total Total with
Average
Standard
DL
no DL
% DL
DL
Deviation of DL
215
88
71%
159
271
229
99
70%
138
267
244
93
72%
117
256
243
125
66%
87
246
163
112
59%
46
132
1094
517
68%
110
245

Max DL
2403
2955
2591
1901
1302
2955

Source: Omniture and World Bank Documents & Reports.

The most downloaded policy reports tend to have a long shelf-life. Some reports with a high number
of downloads experience very high single-day downloads. For example, one report released in FY11,
had 212 downloads on a single day, but then averaged only 3.42 downloads per day during its first year.
Other reports had a high number of first year downloads that were more evenly spread throughout the
time period. Another policy report had an average of 4.14 downloads per day during its first year of
release, yet never had more than 18 downloads in a single day. The most downloaded policy report,
Vietnam Development Report 2009: Capital Matters, received a total of 2,955 downloads, 1,976 of
which were for the English language version of the report and 979 of which were for the Vietnamese
language version. Figure 6 below shows the daily download trends for the five policy reports
(disaggregated by language) with the longest shelf-life; i.e. they had the highest average second-year
downloads in the dataset and were all downloaded on average once a day. It is the case that two of
these reports, were downloaded more often during their second year of release than during their first.
One report was even downloaded more often in its third year of release, 544 times, than in its first two
years combined (500 times).
Figure 6: Download history of selected reports
120
Malaysia economic monitor : repositioning for growth

Daily downloads

100

Investing in a more sustainable Indonesia : country environmental analysis 2009

80

Sudan - The road toward sustainable and broad-based growth

60

Malaysia - Productivity and investment climate assessment update
Vietnam development report 2010 : modern institutions

40
20
0
1

51 101 151 201 251 301 351 401 451 501 551 601 651 701 751 801 851 901 951
Days Since Release (Selected Reports)
Source: World Bank ISPStats and World Bank Documents & Reports.

Citation counts in this study are based on data from Google Scholar. Google Scholar has great breadth
in coverage because it includes not only journal articles, conference proceedings, and other academic

13

reports but also books, working papers, and business and government reports. 19 Google Scholar also
includes non-scholarly citations of articles, which is important since policy reports are not intended for
publication in academic journals which populate typical citation databases such as Web of Science.
Particularly, since objectives of policy reports include informing governments and the development
community, it would not be appropriate to limit the citation count to scholarly databases 20 that exclude
government and think tank publications. Google Scholar gathers bibliographic information by crawling
through websites and fetching HTML and PDF information for the articles, which allows for real-time
updates of citations. 21 Unlike other databases, Google Scholar does not wait for a paper to be formally
published before it fetches its data from working papers and includes it in its database.
Using Google Scholar for citation counts is not without problems. First, its automated software
sometimes does not detect false positives or double-counting. For example, a working paper that has
had different versions uploaded to several repositories could have the citations within each working
paper counted separately, inflating the citation count for these articles. Furthermore, Google Scholar
did not identify those reports that were ultimately published under a new title, either as a book, a
chapter in a book, or as a working paper. Second, it does not have a large coverage of those older, predigital publications unless these publications have been processed with optical character recognition
(OCR) software. Given that our sample only covers fiscal year 2008 through fiscal year 2012 this last
issue should not pose a problem, although there are some policy reports within this period that were
scanned documents converted with OCR software. Third, citations data in Google Scholar is susceptible
to manipulation as any search algorithm would be (Delgado López-Cózar et al., 2012), but it is the
sentiment of the authors that this is not an issue since policy reports are not produced with the main
objective of being cited. Fourth, a report may not receive citations simply because it is not included in
the Google Scholar database. Using data from the World Bank’s Open Knowledge Repository (OKR), we
verified whether the policy reports in our sample were definitely included within the Google Scholar
database. Within our dataset, 410 policy reports were not listed within Google Scholar. 22 We can
assume that these policy reports are not cited, but for the purposes of our analysis we perform
regressions with these missing values removed from the data.
Citation counts are much lower than download counts (Figure 7). Almost 88 percent of the policy
reports (1,054 out of 1,201) in our sample were never cited. 23 Few reports received more than 20

19

It does not include newspaper or magazine articles, blogs, editorials, or reviews.
Prominent sources for citation counts are the Library of Congress’s database, Elsevier’s Scopus database, and
Thomson-Reuter’s Web of Science database. The Library of Congress includes only general bibliographic
information for all types of books and academic articles in their database, and does not have any information on
the citations within the article nor citations of the article. The Scopus and Web of Science databases covers only
serialized publications and conference proceedings, and the journals included in each database differ. Scopus
includes 19,809 journals and Web of Science includes 12,311 across all academic disciplines. Between the two
there are 11,377 shared journals (Center for Research Libraries, 2013). Both include counts of citations by other
articles, but it is limited to citations by articles from within the respective databases.
21
We auto-fetched citation count results using specialized scripts within the open-source program Zotero. There
were constraints to fetching the data based on the nature of Policy Reports: Typically Policy Reports are not
published under individual author’s names, so we exclusively had to rely on the title of the document.
22
Thirty-three reports were not listed in Google Scholar according to OKR, yet had citations within Google Scholar.
These reports were included in the data base.
23
By comparison, Ravallion and Wagstaff (2010) find that nearly 30 percent of World Bank research publications
have never been cited.
20

14

citations. The distribution of citations is skewed like the distribution of downloads and is entered on
zero and low numbers. Of the 147 policy reports that were cited, only 93 were cited between 1 and 5
times. Those policy reports with more than 5 (the “knee of the curve”) citations consist of 54 policy
reports, or 3 percent of the data sample.
Figure 7: Citation Distribution

Citation Distribution of Policy Reports: 2008 to 2012

20
10
0

Number of Reports

30

All Reports with Downloads

0

20

40

60

Downloads per Report
Source: Google Scholar and World Bank Documents & Reports.

Similar to downloads, older policy reports have been cited more often. Over the whole period policy
reports have an average of 1 citation per document, with documents published in 2009 having the
highest citation average of 1.7 citations per document. The citations were collected in the first quarter
of FY13, and have increased since this time. The most cited policy report in our dataset was cited 59
times when the dataset was collected and assembled. As of the third quarter of FY14, the document has
been cited 91 times. 24 Table 4 shows the summary statistics for the cumulative citations for policy
reports.

Fiscal Year
Published
2008
2009
2010
2011
2012
2008–2012

Table 4: Citation Statistics for policy reports
Total in Google Total Total Not
%
Average
Standard
Scholar
Cited
Cited
Cited
Cites
Deviation of Cites
223
36
187
16%
1.32
4.74
242
40
202
17%
1.71
6.94
258
32
226
12%
0.74
2.82
268
26
242
10%
0.77
4.35
210
13
197
6%
0.21
1.50
1201
147
1054
12%
0.96
4.51

Max
Cites
41
59
25
56
20
59

Source: Google Scholar and World Bank Documents & Reports.

24

This policy report is cited more since it was published in the peer-reviewed journal Agricultural Economics.

15

Downloads and citations are count data with a variance that significantly exceeds its mean and a large
number of zeros. Their observations can only have non-negative integer values and they do not have an
explicit upper limit. If the dependent variable is a count variable, the typical econometric regression
tool, the linear regression model, is not appropriate since it is sensitive to both the large number of
zeros and the extreme values that are not uncommon in count data. Assumptions of normality for
count data are difficult to justify unless the data sample is sufficiently large. A more suitable model
would be based on the Poisson distribution, since it specifically models the number of events that occur
over a specific time period, but it works under the assumption that the mean of the count variable is
equal to its variance. But for downloads and citations, the variance significantly exceeds the mean. The
mean for downloads is 110 while the variance is 60,085. 25 A better fit is thus a negative binomial
regression, which is a generalization of the Poisson distribution that includes a parameter to control for
over-dispersion, which leads to confidence intervals that are more precise than those from a Poisson
regression model. It is also appropriate to use in situations where the underlying count process is not
independent 26 (Winkelmann, 2008). Problems with the negative binomial include its low applicability to
data with large numbers of zero observations (Mihaylova et al., 2011).
Besides using the negative binomial model, a second option would be the two-part model, which is
able to account for excess zeros in count data (Winkelmann, 2008). The first part of the model
estimates the probability of the variable being counted (i.e. downloaded or cited), while the second part
estimates the mean number of counts conditional on the count being positive. Logit or probit models
are typically used for the first part, while ordinary least squares, log-linear, or generalized least squares
models are applied for the second. Two-part models appear to outperform other methods when there
are large numbers of zeros in the count data. The results from both models are presented in the report.

V. Results
Our most parsimonious specification shows that costlier reports for middle income countries are
downloaded more. Similar to Wagstaff (2012b), we find that more expensive policy reports tend to have
more downloads. In fact, increasing the budget of a report from around $180,000 (the mean) to around
$316,000 (an increase by half a standard deviation) increases the number of downloads on average by
23 (which is the combined effect of the two part model) or by 30 conditional on the report being
downloaded (the result of the two part model regression). We also include dummies of the year of
disclosure. As expected, reports that have been disclosed for a longer period of time are more likely to
be downloaded (Table B).
Regional reports on larger and richer countries tend to be downloaded more. Anecdotal evidence
suggests that some countries with large populations and middle income country status receive higher
downloads. One would also expect that richer countries are likely to have better internet availability, 27

25

2

The same holds for citations which µ= 0.96 and σ = 20.36.
The existence of contagion or state dependence—that is, the occurrence of an event makes further occurrences
likely—would cause over-dispersion. In the case of downloads, one person’s download is unlikely to be observable
by another person (no data of this kind is provided in D&R), making this possibility unlikely. On the other hand, a
citation by one article is observable by others, and this positive contagion effect could drive the citation count of
policy reports.
27
Though the number of internet users is not significantly linked with increased downloads.
26

16

higher levels of education, and a larger network of university and research centers. We control for these
factors by including variables covering the population and income classification of the country (or
region) covered by the policy report. The base specification also controls for Fund Cost Center Owners
(FCOs). The base FCO dummy is a composite of non-regional and non-network VPUs in the World Bank,
including Legal, Operation Policy and Country Services (OPCS), and World Bank Institute (WBI). For
regional FCOs, the FCO is likely capturing the regional focus of the report. Regional policy reports tend
to receive a higher number of downloads than reports prepared by World Bank networks. In particular,
downloads tend to be particularly high in EAP, ECA, LCR and MNA. Once we control for population and
income classification, all regional dummies, with the exception of EAP, turn insignificant. Population
has a strong effect in generating downloads.
Complex multi-sector and core diagnostic reports are downloaded more frequently. Even when
controlling for costs, multi-sector reports are download more frequently. Multi-sector reports serve as a
proxy for the use of intra-VPU cross support, suggesting that reports which receive intra-VPU cross
support are more likely to be downloaded. Also multi-report, multi-project and core diagnostic reports
tend to be downloaded more frequently conditional on total costs.
The objective of the report matters. As mentioned above, TTLs need to specify the objective of their
report. Within our sample 49 percent of policy reports had the stated objective of informing the public
debate. We find that reports that have the stated objective of informing the public debate are more
likely to be downloaded, and, conditional on being downloaded, are downloaded more frequently.
Dissemination strategies are intended to increase the visibility of reports and, thus, presumably
downloads. In IEG (2008), dissemination efforts for policy reports were among the lowest rated
dimensions of quality 28 according to in-country stakeholders. Sustained follow-up beyond dissemination
seems to be needed, either in the form of lending, from increased funding from other donors for
implementing policy changes, or from training workshops. Government interest also is likely to have a
strong effect on dissemination. In Malaysia, the government led and implemented dissemination efforts
to a much greater effect than the Bank could do alone (IEG, 2008 pg. 58). The IEG report goes on to say
that these examples show that stimulating public debate, particularly debate within the government, is
seen as being important for generating results.
We try to capture dissemination efforts along three dimensions: First, we assess whether policy
reports that were released in the Online Media Briefing Center have higher downloads on average. The
Online Media Briefing Center (OMBC) launches press releases that are available to accredited journalists
before the publication of the report. Within the five years of our data sample, 97 documents were
released in the OMBC. While only 17 policy reports in our dataset were included in the OMBC, the
average downloads per document for these reports were much higher than for those not launched in
OMBC. On average a policy report launched within OMBC had 208 downloads, while a policy report not
launched within OMBC had 109. None of these policy reports pushed by OMBC were core diagnostic
reports. However, once we control for costs, the coefficient on the OMBC dummy, indicating whether
the report has been pushed by OMBC, turns insignificant (Table C). 29 Second, the variable “other costs”

28

The other dimensions were technical quality, relevance, timeliness, and partnership with clients.
Regional and network VPUs also launch press releases, but their data is not centralized and as such is not readily
available. We are thus unable to assess whether policy reports were pushed by their Regional and Network
receive a higher number of downloads.
29

17

could be interpreted as a proxy for dissemination efforts and client engagement, capturing travel costs,
printing costs, etc. In fact, we find that reports which have higher other costs tend to receive more
downloads, conditional on being downloaded. Third, we identified reports that were published in
another type of documents, such as a working paper or a book chapter. We find that the reports were
downloaded at a significantly greater rate in their new incarnation. For example, one policy report was
cited twice within our dataset, but when later published as a working paper with a new title, it was cited
over 50 times.
Citations
A standard measure of an index of research output is the h-index which is highest for reports on Latin
America and the Caribbean followed by the Africa region. The h-index is a measure on the impact of
research output. First proposed in Hirsch (2005), the h-index was meant to quantify an individual’s
scientific output for applications such as faculty recruitment, granting tenure, and awarding grants. An
h-index value of x means that the author has published x items, each of which has been cited at least x
times. It assesses both the productivity and influence of research. As citations tend to increase the
longer a study is published, the h-index tends to rise with years of publications. Figure 8 shows the hindices for the regional VPUs.
Figure 8: H-Indices for the Regions
AFR = 7.00

60

EAP = 7.50

50

ECA = 4.75

40

MNA = 3.00

Citations

LCR = 8.63
SAR = 6.00

30
20
10
0
0

5

10

15

20

25

30

Rank of Policy Report
Source: Google Scholar and World Bank Documents & Reports.
Note: The x-axis ranks individual policy reports by the number of citations. The H-index of
a region is the value of the distance between the x-axis and the region's intersection with
the 45° line.

Contrary to downloads, costs are not a significant determinant of citations (Table D). A more
expensive report is not more likely to be cited or receive a higher number of citations, conditional on
being cited. However, under some specifications, cross support related costs significantly increase the
probability of being cited. The year dummies, which were all significant and positive, indicated that
policy reports that have been disclosed for a longer period of time are cited more frequently. Reports
prepared by network anchors such as HDN and PREM as well as by the Africa and LCR regions tend to
18

have a higher probability of being cited. The number of internet users in a subject country does not
significantly affect citations.
Similar to downloads, more complex reports in larger countries tend to be cited more frequently.
Multi-project, multi-report, and multi-sector reports as well as core diagnostic reports are more likely of
being cited. Also reports on countries with larger populations tend to be cited more frequently. Contrary
to downloads, reports for upper middle countries are not significantly more cited.
Reports pushed by the OMBC received a higher number of citations. The 17 reports that were pushed
by OMBC were cited an average of 7 times, significantly greater than the mean of 0.9 for those reports
not pushed by OMBC (Table E). One of these reports was cited 51 times, potentially benefitting from a
New York Times op-ed by the author. Contrary to downloads, the OMBC dummy remains significant
even when controlling for costs.
Some policy reports may not be cited simply because they were not located in Google scholar. There
are about 410 policy reports that were not cited and not located in Google scholar. Verifying through
other search engines seems to confirm that these reports have not been cited. The results do not
qualitatively change if we were to assume that reports not located in Google scholar have zero citations.
Development objectives do not seem to matter for citations. As expected, we do not find any
systematic evidence that reports with a development objective of informing the public debate receive a
significantly higher or lower number of citations.

VI. Measuring Internal Knowledge Sharing
Measuring internal knowledge transfers is difficult. The key issue is that it is difficult to assess the costs
and, more importantly, the benefits of knowledge sharing among staff because the inputs and outputs
are not systematically monitored and reported, and because of the heterogeneity of the methods of
disseminating knowledge, such as through team-based support, sector-wide support, or individual
training. Two recent papers have tried to assess the demand for and value of research among World
Bank’s operational staff. Ravallion (2011) finds that two-thirds of staff place high value on Bank
research. But it also shows that approximately 23 percent of Bank staff has a low valuation of the
relevance of Bank research for their work, and is uninformed and unfamiliar with its knowledge products.
According to IEG (2012), sector- and anchor-unit based staff rely most often on policy reports from the
anchor units within their own sector, and least often from other units. 30 There is little evidence about
the contribution of cross support to policy reports. This is surprising as some FCOs such as DEC, HDN
Anchor, and PREM Anchor provide more than 8 percent of their staff time to cross support.
In order to efficiently provide knowledge to its external clients, any large international institutions will
have to build effective mechanisms for internal knowledge sharing. When the concept of the World
Bank as a Knowledge Bank was articulated in 1996, networks were created and given the responsibility
to address issues within their fields and to share knowledge with the regions (via sector management

30

Regarding substantial use, 28 percent of staff used Policy Reports from the anchor unit within their own sector,
19 percent used Policy Reports from sector units in other regions, 17 percent from sector units outside their sector,
and 7 percent from DEC.

19

units and cross support). 31 These networks would then provide their knowledge to operational staffs
through cross support.
Cross support is defined as the staff time of an expert purchased from outside the responsible unit for
specific tasks. Cross support can be shared within VPUs and between VPUs, such as a network VPU and
a regional VPU. Cross support across VPUs could thus be used as a measure of internal knowledge
sharing. Cross support is short-term by nature and does not include staff movement or rotation. Cross
support tasks can be both operational and knowledge-generating, and they can take the form of
participation in team visits, preparation of key inputs for analytical and advisory activities (AAA), and
peer review. All cross support costs within FY08–FY12 were extracted from SAP. These data were
matched with the project codes for the policy reports. Those policy reports that did not have matching
cross support data are assumed to have zero cross support costs.
DEC is one of the largest providers of cross-support within the Bank. It has started to provide
increasing cross support over the years, going as far as setting a goal of dedicating a third of its staff
time for this purpose (World Bank, 2011a). In FY12, DEC was the second largest provider of cross
support within the Bank, only exceeded by SDN. Figure 9 below shows the levels of inter-VPU cross
support sold by the different VPU. Although DEC is a large supplier of cross support, key Bank reports
tend to focus on cross support provided by other departments. 32
Figure 9: Total Inter-VPU Cross Support Sold
b) As a Percentage of Staff Years

a) In Staff Weeks

Total Inter-VPU Cross Support Sold, Staff Weeks

10

11.80

9.37
8.37
6.85
6.18

5

Inter-VPU Cross Support Share, percentage

6,000
4,000
2,000

15

Percentage of Staff Years
SDN
PRM
HDN
FPD
SAR
MNA
LCR
ECA
EAP
AFR
WBI
DEC

3.02
1.22 1.20

1.74 1.72 2.02 1.82

SDN
PRM
HDN
FPD
SAR
MNA
LCR
ECA
EAP
AFR
WBI
DEC

0

0

Inter-VPU Cross Support

8,000

Total Inter-VPU Cross Support Sold

2008

2009

2010

2011

2012

FY2008 to FY2012
Source: World Bank Documents & Reports and World Bank SAP database.
Note: Staff are Grade E through Grade I

Source: World Bank SAP database.

DEC also provides the largest amount of cross support by staff and for policy reports. As reported IEG
(2012), in FY10 Bank-wide inter-VPU cross support sold accounted for 5.6 percent of total staff time.
When this is broken down among regions and networks we have 3.3 percent of staff time and 12.5
percent of staff time, 33 respectively. These numbers correspond with calculations made from SAP data,
shown in Figure 10 below. DEC has typically provided the largest share of cross support with levels
around 11.8 percent of total staff years, although the share for cross support provided by PREM has

31

These networks were Human Development, Environmentally and Socially Sustainable Development (now simply
Sustainable Development), Finance, Private Sector, and Infrastructure (all merged into Finance and Private Sector
Development), and Poverty Reduction and Economic Management.
32
See, for example, Annex H of IEG (2012).
33
When calculating staff time, we use the Human Resources definition of a staff year, which is 44 weeks.

20

eclipsed DEC in FY12. Regions have provided very little inter-VPU cross support, although the networks,
especially HDN Anchor, have provided a larger share. DEC is the largest provider of total cross support
for policy reports. Over the past five years DEC has consistently provided the largest amount of cross
support for policy reports, as indicated in the left panel of Figure 10 below. SDN is the second largest
provider, followed the other networks. ECA is the largest regional provider. In the right panel we can
see that most of the cross support from DEC is directed toward AFR and LCR.
Figure 10: Cross Support for Policy Reports
DEC Cross Support to Other VPUs

Inter-VPU Cross Support for Policy Reports

82

0

112 101

117 119 136

FY2008 to FY2012

2,000
1,500

SDN
PRM
HDN
FPD
SAR
MNA
LCR
ECA
EAP
AFR
WBI

1,000

1,062

509 473

500

218

SDN
PRM
HDN
FPD
SAR
MNA
LCR
ECA
EAP
AFR
WBI
DEC

Policy Reports from 2008 through 2012
1,977

381
239

238
106

26

6

1

0

385
355 376

Inter-VPU Cross Support Labor Cost, thousands

664

500

Staff Weeks

1,000

1,103

Source: World Bank SAP database.

Source: World Bank Documents & Reports and World Bank SAP database.

DEC also provides considerable cross support for policy reports as a share of total staff years, but it
was eclipsed by PREM in FY2012 (Figure 11). The shares of cross support for policy reports provided by
DEC equals 24 percent of total inter-VPU cross support. SDN, while a large provider of inter-VPU cross
support, does contribute relatively little to policy reports as a share of total staff weeks.
Figure 11: Cross support for ESW as a share of Total Staff Years
Inter-VPU Cross Support for Policy Reports
3.0
2.0

2.84

1.62

1.0

1.31

0.80
0.60
0.33
0.04 0.06

0.16 0.11 0.19 0.12

0.0

Inter-VPU Cross Support Share, percentage

Percentage of Staff Years

FY2008 to FY2012
Source: World Bank Documents & Reports and World Bank SAP database.
Note: Staff are Grade E through Grade I.

21

SDN
PRM
HDN
FPD
SAR
MNA
LCR
ECA
EAP
AFR
WBI
DEC

DEC cross support is the only VPU linked with greater downloads and greater citations. Cross support
from DEC is significantly associated with more downloads. Cross support provided by FPD significantly
decreases the probability of a report being downloaded as well as the number of downloads (Tables F
and G). No other cross support is significant. Cross support provided by DEC also increase the number of
citations. The relationship between DEC cross support and increased use may also be the manifestation
of selection bias, as TTLs with higher evaluations are more likely to agree to work on policy reports
covering topics of greater interest (and hence more likely to be read and cited).
Reports may receive a higher level of cross support because DEC researchers could cite the reports in
which they participated. Looking at the 25 reports with the highest citations that received DEC cross
support, we find that less than 5 percent of citation were linked to the involved researchers. DEC staff
that provide cross support to a policy report are thus not likely to cite the report later in their own work.

VII. Conclusion
If the objective of a World Bank policy report is to inform the public debate, well-funded, multi-sector
policy reports are likely to do better. More expensive, complex, multi-sector, core diagnostics reports,
such as CEMs or PFRs, on middle-income countries with larger populations tend to be downloaded more
frequently. Those that receive a significant number of downloads tend to have a long shelf-life. There
seems to be a clear demand for multi-sector World Bank reports at a time when the number of
specialized think tanks has increased to over 6,000 (Gann, 2012) and more and more consultancies
engage in providing policy advice to developing countries.
Cross support provided by DEC significantly increases downloads and citation. Very little research has
been performed on the role that Bank-generated research has played in increasing the demand and use
of policy reports. To our knowledge, the only report that explored this link is World Bank (2004), which
investigated the role that knowledge sharing has played in policy report effectiveness. It concluded that
that there was a positive correlation between higher internal quality scores for policy reports and both
the number of DEC research citations in policy reports and the amount of time researchers spent in
preparation of the policy report.
There seems to be some evidence that a media push alone is not sufficient for a good dissemination
strategy. We find some evidence that policy reports that are incarnated in a different format, e.g. as a
World Bank Policy Research Working paper, receive a significantly higher number of downloads. IEG
(2008) provides some discussion that sustained follow-up beyond dissemination is needed, either in the
form of lending, increased funding from other donors for implementing policy changes, or training
workshops to ensure a high visibility of reports. Government interest seems also to have a stronger
effect on dissemination. In Malaysia, the government led and implemented dissemination efforts to a
much greater effect than the Bank could do alone (IEG, 2008, pg. 58). Future research could also assess
which dissemination strategies have been more successful. For example, we are not able to distinguish
whether reports that have a larger dissemination budget are likely to receive a higher visibility from data
currently available in SAP. We can also not assess whether policy reports that were pushed by their
Regional and Network VPUs received a higher number of downloads.
One could also assess different channels through which policy reports could inform the public or
influence the development community or whether reports meet other objectives. First, it would be
interesting to know where reports are actually downloaded. By using IP addresses, it would be possible
22

to identify whether reports tend to be more downloaded in subject countries or, for example, in the
United States, enabling one to track the flow of World Bank knowledge around the world. Second, it
would useful to assess which type of policy reports were most effective in contributing to changes in
government regulation. Third, some World Bank reports may be relevant for other World Bank reports.
World Bank (2004) used citation counts tabulated by Thomas Scientific (later Thomas Reuters) to
demonstrate that those ESWs that more frequently cited Bank research were also more likely to receive
higher QAG quality ratings. This metric is different from what we propose because it looks at what was
cited within a policy report, not how often a policy report was cited. Analyzing how World Bank reports
influence each other could improve our understanding of how internal knowledge spreads within the
institution.

23

References
Center for Research Libraries Academic Database Assessment Tool, last accessed April 28, 2014.
http://adat.crl.edu/.
Delgado López-Cózar, Emilio, Nicolás Robinson-García, Daniel Torres Salinas. 2012. “Manipulating
Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting” EC3 Working
Papers 6:29 May, 2012.
Gann, James G. 2012. “2012 Global Go To Think Tanks Index Report.” Think Tanks and Civil Societies
Program at the University of Pennsylvania. December 2012.
Hirsch, Jorge E. 2005. “And Index to Quantify an Individual’s Scientific Research Output.” Proceedings of
the National Academy of Sciences 102(46): 16569–16572.
Independent Evaluation Group. 2008. Using Knowledge to Improve Development Effectiveness: An
Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006.
Washington DC: World Bank Group.
Independent Evaluation Group. 2012. The Matrix System at Work: An Evaluation of the World Bank’s
Organizational Effectiveness. Washington DC: World Bank Group.
Mihaylova, Borislava, Andrew Briggs, Anthony O’Hagan, and Simon G. Thompson. 2011. “Review of
Statistical Methods for Analysing Healthcare Resources and Costs.” Healthcare Economics. 20:
897–916
Ravallion, Martin. 2011. “Knowledgeable Bankers? The Demand for Research in World Bank Operations.”
World Bank Policy Research Working Paper 5892. December 2011.
Ravallion, Martin and Adam Wagstaff. 2010. “The World Bank’s Publication Record.” World Bank Policy
Research Working Paper 5374. July 2010.
Reeves, Alison. 2011. “Book Review & Media Coverage of World Bank Publications.” Mimeo, World Bank.
March 2011.
Wagstaff, Adam. 2012a. “Who’s writing what in the ‘Knowledge Bank’? And is it being used?” Published
on Let’s Talk Development. http://blogs.worldbank.org/developmenttalk. October 4, 2012.
Wagstaff, Adam. 2012b. “Tracking withdrawals from the ‘Knowledge Bank’” Published on Let’s Talk
Development. http://blogs.worldbank.org/developmenttalk. September 25, 2012.
Winkelmann, Rainer. 2008. Econometric Analysis of Count Data. 5th edition. Berlin: Springer-Verlag.
World Bank. 2000. Poverty Reduction and Global Public Goods: Issues for the World Bank in Supporting
Global Collective Action. September 9, 2000. Washington DC: World Bank Group.

24

World Bank. 2002. Fixing Economic and Sector Work: Update on the Progress of Phase II of the ESW
Reform. Washington DC: World Bank Group.
World Bank. 2004. Report on the World Bank Group Research Program, Fiscal 2002 and 2003.
Washington DC: World Bank Group.
World Bank. 2010a. The World Bank Policy on Access to Information. July 2010. Washington DC: World
Bank Group.
World Bank. 2010b. Performance Assessment Review—World Bank Economic Reports on Growth
Diagnostics in Four Africa Countries: Ghana, Mauritius, Nigeria, and Uganda. Report 55404. June
30, 2010. Washington DC: World Bank Group.
World Bank. 2011a. The State of World Bank Knowledge Services: Knowledge for Development, 2011.
Washington DC: World Bank Group.
World Bank. 2011b. Performance Assessment Review of Investment Climate Assessments in Five
Transforming Economies: Bangladesh, Egypt, Guatemala, Kenya, and Vietnam. Report 62874.
June 30, 2011. Washington DC: World Bank Group.
World Bank. 2012a. The World Bank Quarterly Business and Risk Review Quarter 4 FY12. Washington DC:
World Bank Group.
World Bank. 2012b. Discrete ESW and TA: Guidelines. February 2012. Washington DC: World Bank
Group.
World Bank. 2012c. The World Bank Quarterly Business and Risk Review Quarter 1 FY13. Washington DC:
World Bank Group.
World Bank. 2013a. The World Bank Quarterly Business and Risk Review Quarter 2 FY13. Washington DC:
World Bank Group.
World Bank. 2013b. The World Bank Quarterly Business and Risk Review Quarter 3 FY13. Washington
DC: World Bank Group.
World Bank. 2013c. World Bank Corporate Scorecard: Integrated Results and Performance Framework.
April 2013. Washington DC: World Bank Group.
World Bank. 2013d. Knowledge-Based Country Programs: An Evaluation of the World Bank Group
Experience. July 2013. Washington DC: World Bank Group.

25

VI. Annex

Table A: Data Appendix

Staff Labor Costs

Logarithm of labor costs for all graded Bank staff, not including inter-VPU
cross support, for FY1998 to FY2012. Source: SAP.
Consultant Costs
Logarithm of labor costs for all short-term, extended-term, and vendors,
for FY1998 to FY2012. Source: SAP.
Cross Support
Logarithm of labor costs for inter-VPU cross support (labor provided by a
unit outside the FCO’s VPU), for FY1998 to FY2012. VPUs include DEC, WBI,
AFR, EAP, ECA, LCR, MNA, SAR, FPD, HDN, PREM, and SDN, with an extra
category for other VPUs (OPCS, LEG, etc.). Source: SAP.
Other Costs
Logarithm of costs besides the ones mentioned above. It includes travel,
dissemination, and communication costs, for FY1998 to FY2012. Source:
SAP.
Internet Users
Internet users per 100 people. Source: WDI.
GDP per Capita
Logarithm of the average GDP per capita (PPP constant 2005 international
dollar) for the years 2007 through 2011. Source: WDI.
Population
Logarithm of the average population for the years 2007 through 2011.
Source: WDI.
Multi-Project
Dummy for whether the policy report was financially support by more
than one project codes.
Multi-Report
Dummy for whether the policy report consists of multiple reports.
Multi-Sector
Dummy for whether the policy report was multi-sectoral, as indicated by
positive intra-VPU cross support.
Core Report
Dummy for whether the policy report is a core diagnostic report: Country
Economic Memoranda, Development Policy Reviews, Poverty Assessments
and Public Expenditure Reviews.
CEM
Dummy for whether the policy report is a Country Economic
Memorandum.
OMBC
Dummy for whether policy report was featured within the World Bank’s
Online Media Briefing Center, which launches press releases that are
available to accredited journalists before the publication of the report.
Inform Public Debate
Dummy for whether policy report had objective of informing the public
Dummy
debate. Source: Business Warehouse.
Income Group
Dummies for whether the subject country of the policy report is a low
income, lower middle income, or upper middle income country. Policy
reports that are regional, global, or focus on recent IBRD graduates serve
as the base. Source: World Bank.
Fund Center Owner
Dummies for whether a policy report is owned by one or more of the
following VPUs: AFR, EAP, ECA, LCR, MNA, SAR, FPD, HDN, PREM, and SDN.
Policy reports with multiple project codes could be owned by multiple
VPUs. Other VPUs (WBI, OPCS, LEG, etc.), which own a small number of
policy reports, serve as the base FCO.
Year Dummy
Dummies for whether a policy report was produced in FY2008 through
FY2011; FY2012 is the base year.
Note: The transformation used for the costs and income data was log(x+1). We used this
transformation to maintain observations with zero values.

26

Table B: Regression Results for Downloads, Total Costs
Downloads
VARIABLES
Total Cost
AFR FCO
EAP FCO
ECA FCO
LCR FCO
MNA FCO
SAR FCO
FPD FCO
HDN FCO
PREM FCO
SDN FCO
Year 2008
Year 2009
Year 2010
Year 2011
Multi-Project

(1)
Two Part Model
regress
combined

Negative
Binomial

logit

0.44***
[9.64]
0.26
[0.96]
1.57***
[5.31]
0.75**
[2.53]
0.46
[1.51]
0.52
[1.49]
0.24
[0.80]
-0.18
[-0.51]
-0.15
[-0.32]
0.48
[1.16]
-0.20
[-0.70]
1.80***
[9.51]
1.58***
[8.63]
1.32***
[7.19]
0.76***
[4.31]

0.21***
[4.27]
1.06***
[2.86]
1.44***
[3.75]
1.32***
[3.43]
1.19***
[3.06]
1.16***
[2.73]
0.86**
[2.22]
0.61
[1.47]
0.47
[0.92]
0.44
[0.92]
0.73*
[1.93]
0.59***
[3.20]
0.62***
[3.45]
0.62***
[3.46]
0.32*
[1.87]

51.68***
[6.86]
45.21
[0.99]
246.20***
[5.19]
75.58
[1.57]
61.20
[1.24]
70.23
[1.28]
50.14
[1.01]
2.98
[0.05]
-26.66
[-0.35]
114.90*
[1.79]
26.88
[0.57]
169.49***
[6.02]
147.38***
[5.35]
94.73***
[3.49]
58.03**
[2.15]

-2.64***
[-4.40]

-3.34***
[-4.82]

-651.50***
[-6.50]

1,582
0.0153

1,582
0.0337

1,582

logit

41.26***
[7.81]
61.23*
[1.87]
208.68***
[6.18]
89.33***
[2.61]
75.87**
[2.16]
81.02**
[2.07]
58.69*
[1.66]
19.46
[0.50]
-4.45
[0.08]
90.85**
[1.99]
39.25
[1.16]
132.04***
[6.68]
118.02***
[6.11]
82.19***
[4.31]
48.59**
[2.56]

0.32***
[6.28]
0.20
[0.71]
1.00***
[3.19]
0.49
[1.55]
0.23
[0.67]
0.41
[1.14]
-0.03
[-0.10]
-0.25
[-0.70]
-0.26
[-0.53]
0.17
[0.38]
-0.33
[-1.12]
1.83***
[9.20]
1.62***
[8.71]
1.30***
[6.97]
0.74***
[4.21]
0.40
[1.60]
0.54***
[3.15]
0.39***
[3.01]
0.43**
[2.36]
0.10***
[2.85]
0.23
[0.96]
0.41*
[1.87]
0.51**
[2.43]
-3.40***
[-3.51]

0.11**
[2.04]
0.63
[1.53]
0.96**
[2.27]
0.78*
[1.79]
0.71
[1.62]
0.75
[1.62]
0.52
[1.21]
0.20
[0.45]
-0.20
[-0.35]
-0.12
[-0.22]
0.28
[0.68]
0.60***
[3.19]
0.62***
[3.37]
0.62***
[3.41]
0.31*
[1.78]
1.30***
[3.55]
0.33*
[1.80]
0.24*
[1.90]
0.78***
[3.66]
0.04
[1.05]
-0.02
[-0.08]
-0.08
[-0.37]
0.11
[0.51]
-2.60**
[-2.46]

33.92***
[4.32]
17.21
[0.36]
160.55***
[3.25]
23.09
[0.46]
13.04
[0.25]
28.60
[0.51]
-1.19
[-0.02]
-25.13
[-0.45]
-65.98
[-0.85]
45.31
[0.68]
-15.90
[-0.33]
168.74***
[6.06]
154.74***
[5.69]
90.59***
[3.39]
49.60*
[1.88]
70.77**
[2.15]
90.84***
[3.89]
37.01**
[2.08]
117.06***
[4.91]
17.15***
[3.28]
30.45
[0.88]
79.53**
[2.53]
82.83***
[2.77]
-799.14***
[-5.36]

25.78***
[4.70]
27.56
[0.80]
133.19***
[3.79]
35.11
[0.97]
26.57
[0.72]
38.28
[0.96]
12.09
[0.33]
-12.06
[0.31]
-49.78
[0.91]
27.74
[0.58]
-3.87
[0.11]
129.58***
[6.66]
120.65***
[6.36]
77.10***
[4.14]
41.39**
[2.24]
80.63***
[3.36]
69.92***
[4.24]
31.24**
[2.50]
99.17***
[5.87]
12.63***
[3.43]
20.19
[0.83]
51.96**
[2.36]
59.09***
[2.80]

1582

1,582
0.0183

1,582
0.0569

1,582

1582

Multi-Report
Multi-Sector
Core Report
Population
Low Income
Lower Middle Income
Upper Middle Income
Constant
Observations
Pseudo R-squared
Adj. R-squared

0.128

(2)
Two Part Model
regress
combined

Negative
Binomial

z-statistics in brackets
*** p<0.01, ** p<0.05, * p<0.1

0.177

Table C: Regression Results for Downloads, Cost Components, OMBC & Informing Public Policy
Downloads

(3)
Two Part Model
regress

Negative
Binomial

logit

0.10***
[3.91]
-0.00
[-0.03]
0.02
[1.30]
0.08***
[2.91]
0.49**
[1.97]
0.51***
[2.94]
0.30**
[2.20]
0.44**
[2.46]
0.12***
[3.26]
0.04
[0.14]
0.35
[1.57]
0.37*
[1.73]
0.42
[1.51]
1.04***
[3.32]
0.51
[1.64]
0.32
[1.00]
0.45
[1.25]
0.02
[0.06]
-0.26
[-0.72]
-0.16
[-0.33]
0.17
[0.37]
-0.43
[-1.48]
0.25
[0.46]

0.04*
[1.84]
-0.03
[-1.60]
0.04***
[2.60]
0.04
[1.54]
1.31***
[3.58]
0.37**
[2.00]
0.09
[0.64]
0.78***
[3.61]
0.05
[1.28]
-0.03
[-0.11]
-0.05
[-0.22]
0.08
[0.34]
0.68
[1.63]
1.03**
[2.39]
0.84*
[1.90]
0.82*
[1.85]
0.84*
[1.78]
0.53
[1.22]
0.25
[0.54]
-0.16
[-0.29]
-0.16
[-0.29]
0.31
[0.74]
0.95
[1.21]

12.49***
[3.32]
-0.20
[-0.08]
1.73
[0.93]
5.40
[1.36]
78.78**
[2.40]
91.76***
[3.92]
25.79
[1.38]
118.58***
[4.99]
19.93***
[3.82]
29.74
[0.86]
89.07***
[2.82]
79.57***
[2.65]
31.26
[0.64]
173.83***
[3.49]
31.88
[0.62]
27.69
[0.53]
38.79
[0.69]
1.19
[0.02]
-27.23
[-0.49]
-61.45
[-0.79]
40.86
[0.61]
-20.86
[-0.43]
19.87
[0.29]

Constant

-1.54*
[-1.82]

-2.09**
[-2.20]

-630.62***
[-4.86]

Observations
Pseudo R-squared
Adj. R-squared

1,577
0.0193

1,577
0.0657

1,577

VARIABLES
Staff Labor Costs
Consultant Costs
Cross Support Costs
Other Costs
Multi-Project
Multi-Report
Multi-Sector
Core Report
Population
Low Income
Lower Middle Income
Upper Middle Income
AFR FCO
EAP FCO
ECA FCO
LCR FCO
MNA FCO
SAR FCO
FPD FCO
HDN FCO
PREM FCO
SDN FCO
OMBC
Inform Public Debate

(4)
Two Part Model
regress

Negative
Binomial

logit

9.5***
[3.63]
-0.85
[0.47]
2.04
[1.56]
4.56*
[1.66]
85.4***
[3.59]
71.34***
[4.32]
19.64
[1.50]
99.44***
[5.92]
14.72***
[4.01]
19.57
[0.80]
59.33***
[2.68]
55.94***
[2.65]
37.85
[1.10]
143.13***
[4.06]
41.98
[1.16]
38.7
[1.05]
46.72
[1.17]
13.59
[0.37]
-12.57
[0.32]
-45.74
[0.83]
23.82
[0.50]
-6.71
[0.19]
36.56
[0.72]

0.05*
[1.77]
0.01
[0.39]
-0.01
[-0.49]
0.09***
[3.26]
0.19
[0.80]
0.67***
[3.56]
0.23
[1.62]
0.50***
[2.89]
0.15***
[4.09]
0.19
[0.77]
0.62***
[2.76]
0.54**
[2.43]
0.48
[1.50]
1.13***
[3.24]
0.70**
[2.02]
0.50
[1.38]
0.63
[1.58]
0.16
[0.44]
-0.06
[-0.16]
0.15
[0.26]
0.37
[0.80]
-0.15
[-0.47]

0.03
[0.69]
-0.04
[-1.53]
0.00
[0.13]
0.08**
[2.42]
0.87**
[2.09]
0.62**
[2.29]
0.03
[0.16]
0.69***
[2.71]
0.09**
[2.03]
-0.08
[-0.25]
-0.02
[-0.06]
0.07
[0.24]
0.68
[1.09]
0.67
[1.06]
0.86
[1.32]
0.82
[1.26]
0.67
[0.98]
0.35
[0.55]
-0.32
[-0.50]
-0.16
[-0.19]
0.11
[0.14]
0.71
[1.15]

10.38*
[1.89]
0.75
[0.23]
0.35
[0.16]
8.96*
[1.71]
60.92
[1.60]
124.97***
[4.25]
18.38
[0.82]
130.91***
[4.74]
24.52***
[3.91]
45.66
[1.10]
119.59***
[3.11]
94.05***
[2.58]
64.07
[1.06]
260.24***
[4.13]
86.67
[1.37]
76.32
[1.19]
94.94
[1.35]
60.67
[0.94]
19.48
[0.29]
-11.85
[-0.12]
87.77
[1.11]
-0.69
[-0.01]

8.5**
[2.00]
-0.38
[0.15]
0.32
[0.18]
8.77**
[2.17]
66.91**
[2.19]
109.49***
[4.74]
14.6
[0.83]
115.73***
[5.35]
20.85***
[4.28]
32.76
[1.01]
90.33***
[3.01]
73.08**
[2.56]
64.9
[1.35]
213.57***
[4.26]
86.14*
[1.71]
77.47
[1.52]
87.91
[1.58]
54.39
[1.06]
7.2
[0.13]
-12.79
[0.16]
69.26
[1.11]
16.31
[0.35]

0.28**
[2.14]
-1.82**
[-2.07]

0.14
[0.84]
-2.97**
[-2.39]

42.82*
[1.96]
-811.02***
[-5.07]

35.76**
[2.10]

1577

1,104
0.0182

1,104
0.0752

1,104

1104

combined

0.179

combined

0.196

z-statistics in brackets
*** p<0.01, ** p<0.05, * p<0.1

Note: Year dummies coefficients, which are all similar in magnitude and significance with the results in Table B, are not shown
in order to preserve space.

28

Table D: Regression Results for Citations, Total Costs
Citations
VARIABLES
Total Cost
AFR FCO
EAP FCO
ECA FCO
LCR FCO
MNA FCO
SAR FCO
FPD FCO
HDN FCO
PREM FCO
SDN FCO
Year 2008
Year 2009
Year 2010
Year 2011
Multi-Project

(5)
Two Part Model
regress
combined

Negative
Binomial

logit

0.15
[1.49]
0.47
[0.81]
1.06
[1.62]
-0.28
[-0.45]
1.19*
[1.77]
0.48
[0.55]
0.40
[0.59]
0.46
[0.54]
2.75***
[2.65]
1.26
[1.17]
0.86
[1.38]
2.16***
[4.19]
2.26***
[4.37]
1.67***
[3.23]
1.29**
[2.47]

0.15*
[1.81]
0.47
[1.04]
0.68
[1.44]
0.15
[0.29]
0.87*
[1.78]
0.16
[0.25]
0.56
[1.14]
0.78
[1.40]
2.22***
[3.87]
1.64***
[3.02]
1.01**
[2.26]
1.19***
[3.39]
1.09***
[3.15]
0.83**
[2.37]
0.54
[1.50]

0.78
[1.18]
3.93
[1.17]
4.56
[1.20]
0.18
[0.05]
6.96*
[1.73]
2.74
[0.47]
3.73
[0.92]
0.26
[0.06]
12.79***
[2.69]
4.75
[0.98]
1.92
[0.55]
7.42**
[2.03]
9.49***
[2.67]
5.23
[1.44]
6.02
[1.60]

-4.48***
[-3.13]

-5.27***
[-4.85]

-12.86
[-1.38]

1,182
0.0266

1,182
0.0574

1,182

logit

0.2
[2.00]**
0.83
[1.55]
1.06
[1.81]*
0.13
[0.22]
1.49
[2.41]**
0.45
[0.53]
0.87
[1.41]
0.62
[0.86]
3.2
[4.26]***
1.8
[2.49]**
0.99
[1.81]*
1.79
[3.32]***
1.96
[3.70]***
1.26
[2.39]**
1.13
[2.10]**

0.00
[0.03]
1.20
[1.62]
0.97
[1.23]
0.56
[0.69]
1.51*
[1.82]
0.80
[0.79]
0.54
[0.69]
-0.67
[-0.75]
1.08
[1.04]
0.72
[0.63]
-0.64
[-0.89]
2.91***
[5.48]
2.42***
[4.89]
2.67***
[5.15]
1.35***
[2.69]
1.30**
[2.34]
0.96**
[2.23]
-0.65**
[-2.06]
-1.13**
[-2.32]
0.46***
[4.17]
0.60
[0.90]
-0.80
[-1.37]
0.03
[0.06]
-12.02***
[-4.26]

0.07
[0.79]
1.12**
[2.12]
0.78
[1.45]
0.78
[1.33]
1.19**
[2.11]
0.97
[1.36]
0.64
[1.16]
0.56
[0.91]
1.42**
[2.26]
1.36**
[2.07]
0.21
[0.42]
1.54***
[4.14]
1.24***
[3.42]
1.04***
[2.85]
0.56
[1.49]
0.37
[1.06]
0.65**
[2.46]
-0.06
[-0.28]
-0.78*
[-1.82]
0.33***
[4.00]
0.16
[0.34]
-0.38
[-0.94]
0.19
[0.51]
-10.98***
[-5.19]

0.45
[0.67]
1.06
[0.28]
1.66
[0.41]
-3.48
[-0.80]
4.16
[0.94]
-1.49
[-0.23]
4.64
[1.06]
-1.31
[-0.27]
8.03
[1.62]
-0.33
[-0.06]
-2.31
[-0.61]
9.38***
[2.60]
9.08***
[2.61]
6.12*
[1.68]
5.11
[1.38]
6.64**
[2.15]
4.39*
[1.83]
-3.27
[-1.60]
-2.63
[-0.64]
-0.50
[-0.58]
-1.98
[-0.38]
-8.04*
[-1.78]
0.15
[0.05]
4.66
[0.22]

0.1
[1.00]
0.85
[1.50]
0.7
[1.18]
0.08
[0.13]
1.27
[1.95]*
0.45
[0.50]
0.98
[1.52]
0.21
[0.29]
1.89
[2.58]***
0.84
[1.09]
-0.14
[0.25]
2.13
[4.04]***
1.9
[3.77]***
1.41
[2.76]***
0.98
[1.90]*
1.04
[2.36]**
0.95
[2.74]***
-0.44
[1.51]
-0.82
[1.43]
0.15
[1.29]
-0.13
[0.19]
-1.22
[1.99]**
0.14
[0.30]

1182

1,182
0.0749

1,182
0.144

1,182

1182

Multi-Report
Multi-Sector
Core Report
Population
Low Income
Lower Middle Income
Upper Middle Income
Constant
Observations
Pseudo R-squared
Adj. R-squared

0.0421

(6)
Two Part Model
regress
combined

Negative
Binomial

z-statistics in brackets
*** p<0.01, ** p<0.05, * p<0.1

0.120

Table E: Regression Results for Citations, Cost Components, OMBC & Informing Public Policy
Citations
VARIABLES
Staff Labor Costs
Consultant Costs
Cross Support Costs
Other Costs
Multi-Project
Multi-Report
Multi-Sector
Core Report
Population
Low Income
Lower Middle Income
Upper Middle Income
AFR FCO
EAP FCO
ECA FCO
LCR FCO
MNA FCO
SAR FCO
FPD FCO
HDN FCO
PREM FCO
SDN FCO
OMBC
Inform Public Debate
Constant
Observations
Pseudo R-squared
Adj. R-squared

(7)
Two Part Model
regress
combined

Negative
Binomial

logit

-0.02
[-0.36]
-0.04
[-1.05]
0.07**
[1.99]
-0.08
[-1.29]
1.61***
[3.03]
0.94**
[2.16]
-0.71**
[-2.25]
-1.03**
[-2.18]
0.49***
[4.48]
0.89
[1.34]
-0.53
[-0.92]
0.22
[0.46]
1.19*
[1.65]
0.92
[1.17]
0.56
[0.70]
1.63**
[2.00]
0.80
[0.79]
0.33
[0.43]
-0.38
[-0.43]
1.47
[1.45]
0.46
[0.40]
-0.83
[-1.19]
0.95
[0.95]

-0.03
[-0.70]
0.03
[0.90]
0.06***
[2.60]
-0.09**
[-2.42]
0.46
[1.30]
0.66**
[2.48]
-0.12
[-0.51]
-0.69
[-1.60]
0.33***
[3.92]
0.19
[0.39]
-0.29
[-0.69]
0.30
[0.79]
1.32**
[2.42]
0.92*
[1.67]
0.94
[1.56]
1.34**
[2.31]
0.94
[1.29]
0.77
[1.35]
0.67
[1.03]
1.78***
[2.73]
1.56**
[2.30]
0.36
[0.67]
1.60***
[2.69]

0.13
[0.33]
-0.29
[-1.06]
-0.17
[-0.75]
0.01
[0.03]
7.19**
[2.29]
4.79*
[1.93]
-3.02
[-1.39]
-2.00
[-0.48]
-0.43
[-0.49]
-2.65
[-0.49]
-7.22
[-1.55]
1.01
[0.29]
2.06
[0.53]
1.60
[0.39]
-3.75
[-0.83]
4.42
[0.98]
-2.34
[-0.36]
4.59
[1.02]
-1.30
[-0.26]
8.03
[1.58]
-0.32
[-0.06]
-2.73
[-0.71]
3.84
[0.89]

-11.38***
[-4.49]

-9.69***
[-4.96]

11.73
[0.59]

1,180
0.0806

1,180
0.165
0.106

1,180
0.165
0.106

0
[0.03]
-0.02
[0.46]
0.02
[0.54]
-0.06
[1.12]
1.16***
[2.59]
1***
[2.81]
-0.44
[1.45]
-0.68
[1.18]
0.15
[1.27]
-0.2
[0.28]
-1.05*
[1.68]
0.31
[0.64]
1.08*
[1.84]
0.77
[1.28]
0.14
[0.21]
1.37**
[2.08]
0.31
[0.34]
1.04
[1.59]
0.26
[0.36]
2.09***
[2.80]
0.94
[1.20]
-0.11
[0.18]
1.47**
[2.26]

1180

(8)
Two Part Model
regress
combined

Negative
Binomial

logit

-0.03
[-0.48]
-0.07
[-1.47]
0.04
[1.24]
-0.03
[-0.44]
1.50***
[2.70]
1.11**
[2.28]
-0.81**
[-2.36]
-1.19**
[-2.55]
0.49***
[4.30]
0.83
[1.16]
-0.35
[-0.57]
0.09
[0.17]
1.09
[1.48]
1.05
[1.32]
0.62
[0.75]
1.64**
[1.97]
0.66
[0.62]
0.47
[0.59]
-0.45
[-0.51]
1.68
[1.45]
-0.02
[-0.02]
-1.05
[-1.50]

-0.08*
[-1.71]
0.01
[0.41]
0.04
[1.53]
-0.09**
[-2.13]
0.74**
[2.00]
0.59**
[1.96]
-0.24
[-0.93]
-0.61
[-1.41]
0.34***
[3.89]
0.11
[0.22]
-0.13
[-0.29]
0.14
[0.35]
1.22**
[2.01]
1.08*
[1.68]
1.27*
[1.91]
1.65**
[2.49]
1.06
[1.28]
0.99
[1.49]
0.95
[1.36]
2.10***
[2.63]
1.18
[1.56]
0.73
[1.24]

0.28
[0.69]
-0.20
[-0.66]
-0.09
[-0.36]
0.14
[0.39]
4.71
[1.41]
4.86*
[1.69]
-2.93
[-1.21]
-2.48
[-0.58]
-0.85
[-0.91]
-4.64
[-0.80]
-9.57*
[-1.94]
-0.15
[-0.04]
4.76
[1.16]
2.86
[0.68]
-3.67
[-0.78]
5.86
[1.22]
-0.56
[-0.07]
6.40
[1.34]
0.76
[0.15]
10.70*
[1.96]
2.56
[0.41]
-2.76
[-0.67]

-0.02
[0.33]
-0.02
[0.36]
0.02
[0.42]
-0.05
[0.81]
1.27**
[2.21]
1.17**
[2.38]
-0.62
[1.51]
-0.84
[1.18]
0.14
[0.93]
-0.59
[0.63]
-1.51*
[1.86]
0.09
[0.14]
1.65**
[2.13]
1.27
[1.58]
0.46
[0.53]
2.15**
[2.42]
0.74
[0.58]
1.71*
[1.94]
0.86
[0.92]
3.21***
[3.13]
1.3
[1.18]
0.17
[0.22]

0.21
[0.59]
-11.17***
[-4.21]

0.45*
[1.69]
-9.17***
[-4.40]

-5.89**
[-2.20]
20.76
[0.98]

-0.52
[1.16]

851
0.0827

851
0.185
0.175

851
0.185
0.175

851

z-statistics in brackets
*** p<0.01, ** p<0.05, * p<0.1

Note: Year dummies coefficients, which are all similar in magnitude and significance with the results in Table D, are not shown
in order to preserve space.

30

Table F: Regression Results for Downloads, Cross Support Breakdown
Downloads

(9)
Two Part Model
regress
combined

Negative
Binomial

logit

0.10***
[4.29]
-0.00
[-0.02]
0.09***
[3.59]
0.07***
[3.37]
0.04
[0.97]
-0.04
[-1.04]
-0.01
[-0.14]
0.00
[0.05]
0.05
[1.46]
0.01
[0.33]
0.01
[0.33]
-0.10***
[-3.35]
0.00
[0.15]
0.06**
[2.04]
0.02
[0.98]
0.04*
[1.90]

0.05**
[2.28]
-0.02
[-1.10]
0.06***
[2.64]
0.04*
[1.65]
0.05
[0.97]
-0.03
[-0.85]
0.03
[0.72]
0.06*
[1.65]
0.04
[1.06]
0.04
[0.80]
0.03
[0.72]
-0.02
[-0.82]
0.05
[1.54]
0.05*
[1.82]
0.01
[0.48]
-0.00
[-0.18]

11.88***
[3.13]
2.56
[0.95]
9.92**
[2.44]
8.04***
[2.84]
5.31
[0.87]
-5.57
[-1.06]
-1.36
[-0.25]
-6.64
[-1.61]
8.33*
[1.65]
-4.36
[-0.77]
2.66
[0.50]
-11.55**
[-2.44]
-3.33
[-0.86]
7.47**
[2.06]
0.01
[0.00]
4.40*
[1.69]

Constant

1.25***
[4.39]

-0.70**
[-2.38]

-191.31***
[-3.91]

Observations
Pseudo R-squared
Adj. R-squared

1,580
0.0121

1,580
0.0342

1,580

VARIABLES
Staff Labor Costs
Consultant Costs
Other Costs
DEC Cross Support Costs
WBI Cross Support Costs
AFR Cross Support Costs
EAP Cross Support Costs
ECA Cross Support Costs
LCR Cross Support Costs
MNA Cross Support Costs
SAR Cross Support Costs
FPD Cross Support Costs
HDN Cross Support Costs
PRM Cross Support Costs
SDN Cross Support Costs
Other Cross Support Costs
Multi-Project

logit

9.55***
[3.59]
1.16
[0.61]
8.55***
[3.01]
6.56***
[3.23]
5.16
[1.16]
-4.67
[1.25]
0.02
[0.00]
-2.78
[0.93]
6.93*
[1.91]
-1.88
[0.46]
2.68
[0.71]
-8.6**
[2.57]
-0.8
[0.29]
6.7**
[2.56]
0.31
[0.15]
2.9
[1.56]

0.07***
[2.82]
-0.01
[-0.30]
0.08***
[2.92]
0.06***
[2.98]
0.04
[0.90]
-0.06
[-1.64]
-0.00
[-0.09]
-0.00
[-0.07]
0.04
[0.98]
0.00
[0.05]
0.01
[0.17]
-0.09***
[-2.97]
-0.00
[-0.17]
0.03
[1.11]
0.00
[0.12]
0.02
[1.09]
0.60**
[2.53]
0.61***
[3.48]
0.48***
[3.67]
0.52***
[2.90]
0.09***
[2.89]
0.06
[0.29]
0.48***
[2.59]
0.59***
[3.12]
-0.58
[-0.82]

0.03
[1.27]
-0.03*
[-1.73]
0.05**
[2.06]
0.03
[1.50]
0.04
[0.82]
-0.04
[-1.12]
0.02
[0.37]
0.06
[1.58]
0.04
[0.88]
0.03
[0.75]
0.03
[0.66]
-0.03
[-0.96]
0.04
[1.37]
0.03
[0.91]
0.01
[0.43]
-0.01
[-0.41]
1.29***
[3.69]
0.44**
[2.38]
0.24*
[1.85]
0.76***
[3.55]
0.02
[0.71]
-0.05
[-0.24]
0.02
[0.11]
0.24
[1.17]
-0.99
[-1.32]

9.11**
[2.44]
-0.67
[-0.26]
7.19*
[1.82]
7.89***
[2.91]
5.98
[1.02]
-7.77
[-1.53]
-1.76
[-0.33]
-5.84
[-1.48]
6.28
[1.30]
-3.84
[-0.70]
2.19
[0.43]
-11.86***
[-2.61]
-2.51
[-0.67]
4.16
[1.18]
-0.38
[-0.13]
4.16*
[1.66]
78.93***
[2.62]
112.17***
[4.79]
40.20**
[2.19]
115.55***
[4.85]
22.33***
[4.88]
51.33*
[1.67]
136.07***
[5.00]
130.60***
[4.65]
-647.32***
[-6.03]

6.91***
[2.65]
-1.24
[0.68]
6.11**
[2.22]
6.22***
[3.23]
5.18
[1.23]
-6.29*
[1.76]
-0.78
[0.21]
-2.57
[0.91]
5.17
[1.51]
-1.74
[0.45]
2.17
[0.60]
-8.82***
[2.76]
-0.59
[0.22]
3.53
[1.40]
-0.02
[0.01]
2.64
[1.49]
86.15***
[3.90]
87.3***
[5.28]
33.44***
[2.59]
97.79***
[5.78]
15.78***
[4.89]
33.62
[1.56]
93.09***
[4.85]
94.91***
[4.79]

1580

1,580
0.0179

1,580
0.0602

1,580

1580

Multi-Report
Multi-Sector
Core Report
Population
Low Income
Lower Middle Income
Upper Middle Income

0.0763

(10)
Two Part Model
regress
combined

Negative
Binomial

0.161

z-statistics in brackets
*** p<0.01, ** p<0.05, * p<0.1

Note: Year dummies coefficients are not shown in order to preserve space. As in previous tables, more recent year dummies
have decreasing coefficients and z-statistics.

31

Citations

Table G: Regression Results for Citations, Cross Support Breakdown
(11)
Two Part Model
regress
combined

logit

-0.01
[-0.24]
0.00
[0.12]
-0.04
[-0.77]
0.13**
[2.55]
-0.07
[-0.56]
0.01
[0.12]
0.01
[0.08]
0.12
[1.50]
0.27**
[2.35]
0.03
[0.29]
0.02
[0.20]
-0.04
[-0.56]
-0.01
[-0.10]
-0.16**
[-2.22]
-0.06
[-0.83]

-0.01
[-0.20]
0.03
[1.09]
-0.11***
[-3.03]
0.10***
[3.84]
-0.02
[-0.23]
0.01
[0.15]
0.04
[0.64]
0.08**
[2.13]
0.02
[0.35]
0.05
[0.88]
0.06
[1.22]
-0.03
[-0.50]
-0.00
[-0.05]
-0.02
[-0.49]
-0.03
[-0.83]

-0.08
[-0.22]
-0.21
[-0.72]
0.34
[0.97]
0.43
[1.60]
-0.50
[-0.65]
-0.05
[-0.08]
-0.25
[-0.42]
0.08
[0.20]
0.17
[0.27]
-0.42
[-0.67]
-0.11
[-0.21]
-0.26
[-0.45]
-0.29
[-0.65]
-0.94**
[-2.00]
-0.29
[-0.71]

-0.02
[0.30]
0
[0.05]
-0.04
[0.84]
0.13***
[3.30]
-0.07
[0.68]
0
[0.00]
0
[0.01]
0.08
[1.29]
0.04
[0.41]
-0.01
[0.11]
0.04
[0.50]
-0.05
[0.64]
-0.04
[0.58]
-0.13*
[1.94]
-0.06
[1.03]

-0.04
[-0.61]
-0.05
[-1.36]
-0.04
[-0.62]
0.15***
[3.47]
-0.11
[-0.88]
-0.02
[-0.24]
-0.08
[-0.76]
0.12*
[1.80]
-0.05
[-0.53]
0.13
[1.08]
0.11
[1.39]
-0.03
[-0.36]
0.07
[0.99]
-0.10
[-1.40]
0.00
[0.02]

-0.04
[-0.98]
0.01
[0.42]
-0.07*
[-1.94]
0.10***
[3.50]
0.01
[0.07]
-0.02
[-0.38]
0.02
[0.40]
0.12***
[2.77]
-0.01
[-0.13]
0.04
[0.61]
0.05
[0.93]
-0.05
[-0.76]
0.06
[1.26]
-0.02
[-0.34]
-0.00
[-0.09]

-0.04
[-0.11]
-0.19
[-0.69]
0.08
[0.23]
0.46*
[1.81]
-0.91
[-1.21]
-0.27
[-0.45]
-0.44
[-0.76]
0.33
[0.83]
-0.52
[-0.86]
-0.74
[-1.22]
0.06
[0.11]
-0.05
[-0.08]
-0.11
[-0.25]
-0.87*
[-1.90]
-0.29
[-0.76]

-0.03
[0.58]
-0.01
[0.37]
-0.04
[0.76]
0.12***
[3.27]
-0.11
[1.01]
-0.05
[0.57]
-0.04
[0.46]
0.12**
[2.09]
-0.07
[0.82]
-0.06
[0.74]
0.04
[0.57]
-0.04
[0.45]
0.02
[0.40]
-0.12*
[1.80]
-0.04
[0.71]

-0.08
[-1.38]

-0.01
[-0.28]

0.13
[0.40]

0.01
[0.21]

Constant

-1.51**
[-2.02]

-2.17***
[-4.21]

2.71
[0.54]

-0.01
[-0.25]
1.86***
[3.49]
0.91**
[1.99]
-0.58*
[-1.86]
-1.05**
[-2.04]
0.41***
[4.43]
0.83
[1.47]
-0.70
[-1.41]
0.51
[1.10]
-9.11***
[-4.49]

-0.01
[-0.27]
0.91***
[2.92]
0.75***
[2.85]
-0.05
[-0.21]
-0.61
[-1.45]
0.27***
[4.30]
0.02
[0.04]
-0.59*
[-1.79]
0.13
[0.39]
-7.42***
[-5.23]

0.43
[1.33]
7.81***
[2.76]
4.51*
[1.73]
-3.09
[-1.42]
-3.78
[-0.89]
0.00
[0.00]
1.66
[0.38]
-6.03*
[-1.67]
2.93
[0.93]
1.57
[0.11]

0.05
[1.04]
1.54***
[3.73]
1.04***
[2.80]
-0.41
[1.33]
-0.86
[1.46]
0.18*
[1.93]
0.21
[0.36]
-1.12**
[2.25]
0.44
[1.00]

Observations
Pseudo R-squared
Adj. R-squared

1,181
0.0235

1,181
0.0543

1,181

1181

1,181
0.0801

1,181
0.160

1,181

1181

VARIABLES
Staff Labor Costs
Consultant Costs
Other Costs
DEC Cross Support Costs
WBI Cross Support Costs
AFR Cross Support Costs
EAP Cross Support Costs
ECA Cross Support Costs
LCR Cross Support Costs
MNA Cross Support Costs
SAR Cross Support Costs
FPD Cross Support Costs
HDN Cross Support Costs
PRM Cross Support Costs
SDN Cross Support Costs
Other Cross Support
Costs
Multi-Project
Multi-Report
Multi-Sector
Core Report
Population
Low Income
Lower Middle Income
Upper Middle Income

-0.0362

Negative
Binomial

(12)
Two Part Model
logit
regress
combined

Negative
Binomial

z-statistics in brackets
*** p<0.01, ** p<0.05, * p<0.1

0.0851

Note: Year dummies coefficients are not shown in order to preserve space. As in previous tables, more recent year dummies
have decreasing coefficients and z-statistics.

32


Documentos relacionados


Documento PDF wps6851
Documento PDF analyzing soviet defense programs 1951 1990
Documento PDF the economics of world war ii
Documento PDF paviolo et al 2006 the need of transboundary effort to conserve jaguars cat news 45 061108
Documento PDF db14 full report
Documento PDF resolution wpstates 1983 01 eng


Palabras claves relacionadas