Thursday, March 29, 2018

Launch of “The Cultural Revolution in Images: Caricature-Posters from Guangzhou 1966-1977”

The Library is proud to partner with  Dr. Laura Pozzi , Lecturer of the   Centre for China Studies , to launch this new digital colle... thumbnail 1 summary

The Library is proud to partner with Dr. Laura Pozzi, Lecturer of the Centre for China Studies, to launch this new digital collection, “The Cultural Revolution in Images: Caricature-Posters from Guangzhou 1966-1977”. Through high-definition digitization, the collection showcasing 216 original posters from the Library on the Cultural Revolution is now available online to the academic community throughout the world. With the curation effort of Dr. Pozzi, all the posters were meticulously classified and the collection is organized into two sections: the first entitled “Caricature Posters of the Cultural Revolution in Guangzhou” showcases posters dated between 1966 and 1967; the second section entitled “The End of the Gang of Four” presents posters dated between 1976 and 1977. The posters are then divided according to their content, mostly the identity of the politician they targeted. All the information about each image (author and date of production) together with a translation of texts and slogans contained in the posters are accessible by either clicking the individual poster or mouse over an individual poster in each section. 
According to Dr. Pozzi, most of the pieces showcased in this collection are satirical caricatures against party leaders drawn in Guangzhou during the Cultural Revolution. This satirical content sets them apart from most known images of the Cultural Revolution, which are mainly propagandistic. It is hoped that this unique and valuable collection will be helpful to scholars in their research and teaching.
The digital collection can be accessed at: http://repository.lib.cuhk.edu.hk/en/collection/crposter
For enquiries, please contact us at digitization@lib.cuhk.edu.hk
香港中文大學圖書館很榮幸與中國研究中心講師 Laura Pozzi博士合作推出新的圖書館數碼館藏:”漫畫中的文化大革命:廣州的諷刺宣傳畫 1966-1977”。216幅文化大革命期間的政治諷刺宣傳畫全部由圖書館收藏,透過高清掃瞄將此數碼館藏呈現給全世界。館藏中所有海報均由Pozzi博士精心分類為兩部分:第一部分“文革期間的廣州諷刺宣傳畫”收錄了1966至1967年的宣傳畫;第二部分“粉碎四人幫”則收錄1967至1977年的作品。除了以年份為主要區分,宣傳畫亦依照內容,尤其是作品中政治人物再作分類。讀者可將鼠標移至宣傳畫海報及點擊海報以了解更多相關資料,當中包括作者及創作年份及宣傳畫中所使用中文文本及口號的英譯版本。
根據Pozzi博士介紹,此數碼館藏的大部分作品都是文革期間廣州民眾針對各黨派領袖所繪畫的諷刺漫畫。當中內容與黨派領袖在文化大革命中所宣傳的形像大相徑庭。本館衷心希望此獨特而有價值的藏品有助於學者進行研究及教學。
如有查詢,可電郵至 digitization@lib.cuhk.edu.hk 與我們聯絡。



Wednesday, March 14, 2018

Innovation Skills Workshops: Applying Digital Tools in Telling Stories with DATA (First Workshop on 23-24 March 2018)

The Library is pleased to collaborate with  Centre for Entrepreneurship  in arranging a series of workshops -  Innovation Skills Workshops:... thumbnail 1 summary
The Library is pleased to collaborate with Centre for Entrepreneurship in arranging a series of workshops - Innovation Skills Workshops: Applying Digital Tools in Telling Stories with DATA from March to October 2018. The series consists of 3 workshops well-structured for equipping participants with the skills to combine computational methods with narrative approaches using data to develop web applications for scholarly and creative works. Popular digital tools like Python will be covered. Each workshop consists of 9 hours with theories and practices, starting on Friday night and end on Saturday afternoon:
  • Workshop 1: Design Thinking Meets Computational Thinking - Digital Literacy in the Network Age
  • Workshop 2: Preparing and Exploring Your Data in Python
  • Workshop 3: Visualizing and Publishing Your Data in Python
Welcome CUHK Faculty members, researchers and all students to join this workshop to equip yourself to tell impactful stories with data. Please also keep an eye on the other two workshops. Certificate of Attendance will be issued for participants who have attended ALL THREE workshops.
Workshop 1: Design Thinking Meets Computational Thinking - Digital Literacy in the Network Age (9 hrs) (will be held on 23-24 March 2018 (Fri–Sat)) 
a. T-shaped Talent and Digital Literacy
  1. From I-shaped to T-shaped: Talent development for the network age
  2. Design Thinking meets Computational Thinking: A STEAM approach to digital literacy
  3. Lessons from “Digital Humanities”: C.P. Snow, Nicholas Negroponte, and Lev Manovich revisited
  4. Telling story with data: From data scraping to data visualisation and interaction
Date & Time: 23 March 2018 (Fri), 6:30 p.m. – 9:30 p.m.
b. The Big 3 of web publishing
  1. HTML - the noun in web publishing
  2. CSS - the adjective in web publishing
  3. JavaScript - the verb in web publishing
  4. Using Git, Bootstrap library and Pingendo Builder for web development and publishing
Date & Time: 24 March 2018 (Sat), 10:00 a.m. – 1:00 p.m. & 2:30 p.m. - 5:30 p.m.
Workshop 2: Preparing and Exploring Your Data in Python (9 hrs) (tentatively in mid-May to early June 2018) 
a. Preparing (pre-processing) your data for growth (3 hrs)
  1. Know your sources: interviews, field studies, open data, API, websites, IoT, and digital archives
  2. ETL (extraction, transformation, and loading) in CSV, XML, and JSON formats for data preparation
  3. Finding a home for your data - cloud computing and its infrastructure for growth and support
  4. Popular tools for data preparation (e.g. Knime, Open Refine, Google Sheet/Xpath, Scrapinghub, Beautiful Soap, and Scrapy)
b. Exploring your data in Python (6 hrs)
  1. Using Anaconda Jupyter Notebook for data exploration in Python
  2. Introduction to Python operations (operator and operand), control structure, data structure, and function
  3. Useful Python modules for data exploration, analysis and mining (Mathpotlib, Numpy, Pandas, etc.)
  4. Free online resources for self-paced learning in Python (codeacademy.com, coursera.org, udacity.com, cognitiveclass.ai, etc.)
Workshop 3: Visualizing and Publishing Your Data in Python (9 hrs) (tentatively in mid-September to early October 2018) 
a. Growing your data in the cloud: From Google Sheet to Airtable (3 hrs)
  1. Beyond Google Sheet — Building relational database in Airtable for storing and managing your data
  2. The power of views — Displaying and filtering data in form, grid, calendar, kanban, and gallery views
  3. Functions and API for more advanced data modelling and application development
  4. Integration with other web applications for team collaboration and project management
b. Data Visualisation in JavaScript (3 hrs)
  1. Front-end vs. back-end programming: interface with the user and interface with the data using the Python Flask framework
  2. Useful JavaScript libraries (jQuery, D3, Mpld3, Leaflet, etc.) for data visualization and front-end interactions
  3. Create your first interactive chart in Matpotlib and Mpld3 
  4. Create your first interactive map in Leaflet
c. Publishing Your Project on the Web (3 hrs)
  1. The elements of user experience in web design
  2. The narrative components in user journey within a web design
  3. Combining Airtable and Bootstrap library for web publishing
  4. Use of Google Optimize and Google Analytics to track your web project
Venue: Digital Scholarship Lab, G/F, University Library
Registration: Click to register (for workshop 1)
Remarks: Users are required to bring their own devices to the workshop.
Enquiries: dslab@lib.cuhk.edu.hk.
Innovation Skills Workshop: Applying Digital Tools in Telling Stories with DATA

Monday, January 22, 2018

First Collaborative Digital Scholarship Project Launched

The first Digital Scholarship Project in collaboration with Prof Celine Lai of Faculty of Arts has been soft launched together with the Digi... thumbnail 1 summary
The first Digital Scholarship Project in collaboration with Prof Celine Lai of Faculty of Arts has been soft launched together with the Digital Scholarship Projects website in early January 2018.

The project "GIS Mapping and Archaeology of Early China" was collaborated with Prof. Celine Lai of Faculty of Arts in using GIS for mapping archaeological sites embedded with unearthed bronzes details. The data originally in excel files was collected from her doctoral study on the topic and at the initiation of Prof. Lai, dynamic maps are employed to visualise the distribution of the archaeological sites for people who are interested to the topic to further study.

To enable researchers to make use of the research data of Prof. Lai and to avoid re-inventing the wheel, the project web site also provides downloadable files in three different formats with the full list of information and references on the bronzes and archaeological sites for producing the map.  
  • PDF file of the table for full references and quotes of information
  • Files for visualising the maps and for further analysis:
    • Shapefile for use in GIS software
    • kmz file for use in Google Map / Google Earth
Please visit the project website: http://dsprojects.lib.cuhk.edu.hk/en/projects/gis-nao/ for these files and more information.  The project will continue as more data are provided by Prof. Lai.

Apart from this collaborative project, the DS Service Team has also released another small project: Text Data Preparation: A Practice in R using the Sheng Xuanhuai Collection in demonstrating the use of R in preparing data for Chinese text-mining and analysis.  Future Digital Scholarship Projects will all be available in this Digital Scholarship Projects website.


About Digital Scholarship Service in CUHK Library
The Digital Scholarship Service Team aims on collaborating with CUHK Faculty members, researchers and postgraduates in facilitating digital scholarship research.  CUHK community are welcome to visit our Digital Scholarship Services website or email us at dslab@lib.cuhk.edu.hk if your are interested to know more.

Conference: IP Protection and Enforcement: from Policy to Strategy to Practice

Faculty members and postgraduate students are increasingly aware of the copyright issues in teaching, learning, and research. Copyright is ... thumbnail 1 summary

Faculty members and postgraduate students are increasingly aware of the copyright issues in teaching, learning, and research. Copyright is only part of Intellectual Property that also covers other intangible property right such as trade marks, patents, designs, plant varieties and the layout design of integrated circuits. Intellectual property protection is important to Hong Kong being an international trading centre. If you want to know more about the IP policy and current practices, you are welcome to join the conference on IP Protection and Enforcement: from Policy to Strategy to Practice organized by Faculty of Law, CUHK

The conference has two major focuses. The first session focuses on the enforcement practices and IP initiatives by the Government of the Hong Kong SAR. The speakers are top officials from the Intellectual Property Department, Customs and Excise Department, and Hong Kong Trade Development Council. The second session focuses on the best enforcement practices in the business sector. The speakers are industry leaders and top IP lawyers. 



Date: 26 Jan 2018 (Fri)
Time: 9:00 a.m. – 1:40 p.m.
Venue: The CUHK Graduate Law Centre, 2/F Bank of America Tower, Central
Conference programme: http://www.law.cuhk.edu.hk/en/event-page/file/20180126_Event_agenda.pdf

Digital Scholarship Consultation Hours in University Library Continues From 16 January, 2018

Digital Scholarship Consultation Hours service has been commenced since  October 2017 to provide instant discussion and advice to any walk... thumbnail 1 summary
Digital Scholarship Consultation Hours service has been commenced since October 2017 to provide instant discussion and advice to any walk-in users with questions on:

  • Using the facilities of the Digital Scholarship Lab
  • Software and tools in the Lab such as ArcGIS, Gephi, Omeka and Voyant Tools
  • Locating data sources
  • Applying digital scholarship research methodologies and visualising data
  • Copyright issues on the use of research data and materials.
On every alternate Tuesday, the Digital Scholarship team does have interesting in-depth discussion and demonstration with academics and postgraduates that come to the Lab. The service will continue in this second semester starting from 16 January 2018 (Tue) on every alternate Tuesday during term time:

Date: 16 & 30 January
        13 & 27 February
        13 & 27 March
        10 & 24 April

Time: 3:00 – 5:00 p.m.

Venue: Digital Scholarship Lab, G/F, University Library

CUHK colleagues and students are welcome to visit us for an instant discussion of your research, or email to dslab@lib.cuhk.edu.hk your question for help!



Thursday, December 28, 2017

Text Data Preparation: a Practice in R using the Sheng Xuanhuai Collection

In this post, we share a way of preparing Chinese text data for computational analysis; we do so in R using sample texts from a historical... thumbnail 1 summary
In this post, we share a way of preparing Chinese text data for computational analysis; we do so in R using sample texts from a historical collection that is currently being digitized by our library - the Sheng Xuanhuai Collection.

The Sheng Xuanhuai collection contains over 70 volumes of correspondences between the entrepreneur Sheng Xuanhuai and other individuals. The texts included in the collection are digitized and are preserved in formats of images and text files. The texts are also coded with labels/variables such as title, sender name, receiver name, date, key words and locations mentioned in the texts. The digitization and transcription of the correspondences that transform these texts into machine readable formats allows researchers to conduct studies using computational text analysis and other relevant methods.

In the following sections, we demonstrate our way of importing text data to R, preparing texts for analysis, as well as exploring and visualizing texts. Basic knowledge of R will be helpful if you are to try this practice or even apply this on your own data.

Import Text

First we need to read our data (the texts - csv files are used in this demo) to R - to do so, we use setwd() function to set up the working directory, i.e., let R know the path of where we store the data on the computer, then we use read.csv() to load the data file named v36.csv to R.
# set your working directory
setwd('YOUR WORKING DIRECTORY')
# load the data in a spreadsheet to R
v36 <- read.csv('v36.csv', encoding = 'UTF-8', header=TRUE, row.names=1, stringsAsFactors=FALSE)

# view the first two rows of the data
head(v36, 2)
##                lid
## 1 36-001A-1—001A-2
## 2 36-001B-1—001B-2
##                                                                                                                                                                                                                                                                                                           ltext
## 1                                                              宮保鈞座:新正趨賀兩次,聞藎躬尚未康復,不可以風,未敢驚擾。昨往拜朱邇典。伊面稱久違侍鈞顔,欲於日內晉謁,兼答拜經方。頃接其來函,准於明日下午三句鐘來署拜謁。可否允其接見?謹請示復,以便函告該使。肅此。上叩勛祺,順頌新禧。經方謹上。初九日。
## 2 啟者:頃據民政部公啟稱:於本月十六日下午三點鐘,在部開第一次衛生會會議。請飭派出各員届時蒞會等因。査本部知醫者惟屈道永秋( 去年九月始行札調到部),及承政廳行走學部主事謝天保兩員。屈道在本部月支薪水二百兩,謝主事則於滬甯鐵路掛名,月支洋百元。此兩員在本部無事,可否派其前往會議?請酌示。經方謹上。十二日。
There are two variables in the data: lid and ltext - they are correspondence letter ID and letter text in volume 36 of the collection. We have 245 rows in this dataset, i.e., 245 letters in this volume.

Segmenting Chinese Text

Words and terms are the basic units of many computational text analysis methods, however Chinese characters are not “naturally” divided by whitespaces like some other languages such as English. A number of methods are developed to segment Chinese characters - here we try the widely used “jieba” segmenter on our sample texts. To use the R version of jieba, install the package by running this command install.packages('jiebaR') in you R Console. Note you also need to run install.packages() for the other packages we use here in the following sections if you haven’t had them installed on your computer.
# load the "jiebaR" and "stringr" packages/libraries
library(jiebaR)
library(stringr)
Initialize an engine for word segmentation, use all the default settings, and try it on a simple sentence.
# initialize jiebaR worker
cutter <- worker()

# test the worker
cutter["今天的天氣真好"]
## [1] "今天" "的"   "天氣" "真好"
We then define a function called seg_x by which we segment the texts stored in the ltext variable of the data v36 and save them as a new variable of v36 called ltext.seg.
# define the function of segmenting
seg_x <- function(x) {str_c(cutter[x], collapse = " ")} 

# apply the function to each document (row of ltext)
x.out <- sapply(v36$ltext, seg_x, USE.NAMES = FALSE)

# attach the segmented text back to the data frame
v36$ltext.seg <- x.out 

# view the first two rows of the data frame
head(v36, 2)
##                lid
## 1 36-001A-1—001A-2
## 2 36-001B-1—001B-2
##                                                                                                                                                                                                                                                                                                           ltext
## 1                                                              宮保鈞座:新正趨賀兩次,聞藎躬尚未康復,不可以風,未敢驚擾。昨往拜朱邇典。伊面稱久違侍鈞顔,欲於日內晉謁,兼答拜經方。頃接其來函,准於明日下午三句鐘來署拜謁。可否允其接見?謹請示復,以便函告該使。肅此。上叩勛祺,順頌新禧。經方謹上。初九日。
## 2 啟者:頃據民政部公啟稱:於本月十六日下午三點鐘,在部開第一次衛生會會議。請飭派出各員届時蒞會等因。査本部知醫者惟屈道永秋( 去年九月始行札調到部),及承政廳行走學部主事謝天保兩員。屈道在本部月支薪水二百兩,謝主事則於滬甯鐵路掛名,月支洋百元。此兩員在本部無事,可否派其前往會議?請酌示。經方謹上。十二日。
##                                                                                                                                                                                                                                                                                                                                                 ltext.seg
## 1                                                                               宮保 鈞座 新 正 趨 賀 兩次 聞 藎 躬 尚未 康復 不 可以 風 未敢 驚擾 昨往 拜 朱邇 典 伊面 稱 久違 侍鈞 顔 欲 於 日 內 晉謁 兼 答拜 經方 頃接 其 來函 准 於 明日 下午 三句 鐘來署 拜謁 可否 允其 接見 謹 請示 復 以便 函告 該 使 肅此 上叩 勛 祺 順頌 新禧 經方謹 上 初九 日
## 2 啟者 頃 據 民政部 公 啟稱 於 本月 十六日 下午 三點鐘 在 部開 第一次 衛生 會 會議 請 飭 派出 各員届 時 蒞會 等 因 査 本部 知 醫者 惟屈 道 永秋 去年 九月 始行 札 調到 部 及承政廳 行走 學部 主事謝 天保 兩員 屈道 在 本部 月 支 薪水 二百兩 謝主事則 於 滬 甯 鐵路 掛名 月 支洋 百元 此 兩員 在 本部 無事 可否 派 其 前往 會議 請 酌 示 經方謹 上 十二日

Create corpus and document-term/feature-matrix

With the texts segmented by whitespaces, we can move on to create corpus and document-term/feature-matrix (DTM/DFM) that are often used for further text analysis. Here we use functions of the quanteda package to create corpus and DFMs, so does to explore and visualize the texts. quanteda is an R package for managing and analyzing text data; it provides tools for corpus management, natural language processing, document-feature-matrix analysis and more.
# load the library
library(quanteda)
We create a corpus from the texts stored in the ltext.seg variable using the corpus() function. We also tokenize the texts using tokens() and construct a document-feature-matrix using dfm(). Note “fasterword” is specified so that the texts are tokenized by whitespaces. We can then view the most frequent terms/features in this set of texts using topfeatures(). The quanteda package also offers a function textplot_wordcloud() by which you can easily plot a wordcloud from DFMs.
# create corpus
lcorpus <- corpus(v36$ltext.seg)
# summarize the lcorpus object
summary(lcorpus, showmeta = TRUE, 5)
## Corpus consisting of 245 documents, showing 5 documents:
## 
##   Text Types Tokens Sentences
##  text1    73     82         1
##  text2    82    101         1
##  text3   121    153         1
##  text4    64     70         1
##  text5   171    298         1
## 
## Source:  /Users/Guest/Desktop/sheng/* on x86_64 by Guest
## Created: Wed Dec 27 15:14:39 2017
## Notes:
# see the text in the 1st document of lcorpus
texts(lcorpus)[1]
##                                                                                                                                                                                                                                                                       text1 
## "宮保 鈞座 新 正 趨 賀 兩次 聞 藎 躬 尚未 康復 不 可以 風 未敢 驚擾 昨往 拜 朱邇 典 伊面 稱 久違 侍鈞 顔 欲 於 日 內 晉謁 兼 答拜 經方 頃接 其 來函 准 於 明日 下午 三句 鐘來署 拜謁 可否 允其 接見 謹 請示 復 以便 函告 該 使 肅此 上叩 勛 祺 順頌 新禧 經方謹 上 初九 日"
# create dfm with "terms/features" spliting by whitespaces; 
# ie, preserve what has done for segmenting by jiebaR

# tokenize:"tokens" from doc 1, split by whitespaces
tokens(lcorpus, what = "fasterword")[1]
## tokens from 1 document.
## text1 :
##  [1] "宮保"   "鈞座"   "新"     "正"     "趨"     "賀"     "兩次"  
##  [8] "聞"     "藎"     "躬"     "尚未"   "康復"   "不"     "可以"  
## [15] "風"     "未敢"   "驚擾"   "昨往"   "拜"     "朱邇"   "典"    
## [22] "伊面"   "稱"     "久違"   "侍鈞"   "顔"     "欲"     "於"    
## [29] "日"     "內"     "晉謁"   "兼"     "答拜"   "經方"   "頃接"  
## [36] "其"     "來函"   "准"     "於"     "明日"   "下午"   "三句"  
## [43] "鐘來署" "拜謁"   "可否"   "允其"   "接見"   "謹"     "請示"  
## [50] "復"     "以便"   "函告"   "該"     "使"     "肅此"   "上叩"  
## [57] "勛"     "祺"     "順頌"   "新禧"   "經方謹" "上"     "初九"  
## [64] "日"
# tokenize and create document-feature-matrix
ltokens <- tokens(v36$ltext.seg, what = "fasterword")
ldfm <- dfm(ltokens)

# a dfm with 245 documents and 8052 features
ldfm 
## Document-feature matrix of: 245 documents, 8,052 features (98.9% sparse).
# list top 20 features
topfeatures(ldfm, 20)
##     上   宮保   鈞座 經方謹     之   肅頌     復     與     已   崇綏 
##    216    194    176    171    168    149    147    134    134    130 
##     呈     為     又     在     於     請     附     係     以     有 
##    122    109    107    105     97     87     86     86     85     85
# plot wordcloud
par(family='Kaiti TC') # set Chinese font on Mac; you may not need to set font on Windows
textplot_wordcloud(ldfm, min.freq=30, random.order=FALSE,
                   colors = RColorBrewer::brewer.pal(8,"Dark2"))



Combine multiple data files

In the above lines we show how to work with texts stored in one single file, however it is also fairly common that we have texts saved in multiple files. Here we demonstrate how we combine more than one text file in a more efficient way than do it one by one and also some more ways and options of segmenting, exploring and visualizing text data.
Let’s start fresh by removing what we have loaded and created in R.
# remove everything in R environment
rm(list=ls())

We first define a function named multcomb to do the following: 1) list the file names of all the data files that you would like to combine to one file - in this case, we have two csv files to combine, 2) read in the files one by one and rbind them to one data frame.
Save all the data files in one folder, then plug in the path of the folder in the multcomb function to combine all the data files - here we save the combined data frame as mydata.
# define the function of combining multiple files
multcomb <- function(mypath){
  # save all the file names (with path) in an object "filenames"
  filenames <- list.files(path=mypath, full.names=TRUE)
  # import all files and save them as "datalist"
  datalist <- lapply(filenames, function(x){
    read.csv(file=x, encoding='UTF-8', header=TRUE, row.names=1, stringsAsFactors=FALSE)})
  # combine the files (data frames in "datalist")
  Reduce(function(x,y) {rbind(x,y)}, datalist)}
# Use the function multcomb to combine the files in the folder:
# before excecute the function, save all the csv. files in one folder;
# note the folder should not contain other files
mydata <- multcomb('YOUR PATH OF THE FOLDER')
# view the first two rows of mydata
head(mydata, 2)
##                lid
## 1 36-001A-1—001A-2
## 2 36-001B-1—001B-2
##                                                                                                                                                                                                                                                                                                           ltext
## 1                                                              宮保鈞座:新正趨賀兩次,聞藎躬尚未康復,不可以風,未敢驚擾。昨往拜朱邇典。伊面稱久違侍鈞顔,欲於日內晉謁,兼答拜經方。頃接其來函,准於明日下午三句鐘來署拜謁。可否允其接見?謹請示復,以便函告該使。肅此。上叩勛祺,順頌新禧。經方謹上。初九日。
## 2 啟者:頃據民政部公啟稱:於本月十六日下午三點鐘,在部開第一次衛生會會議。請飭派出各員届時蒞會等因。査本部知醫者惟屈道永秋( 去年九月始行札調到部),及承政廳行走學部主事謝天保兩員。屈道在本部月支薪水二百兩,謝主事則於滬甯鐵路掛名,月支洋百元。此兩員在本部無事,可否派其前往會議?請酌示。經方謹上。十二日。

Segmenting: stopwords and dictionary

Segment the words in the combined data file - this time we use stopwords and dictionary to modify the “worker” of segmenting.
# see the stopwords and dictionary
readLines('sheng_stop.txt', encoding = 'UTF-8')
##  [1] ""   "之" "與" "為" "也" "有" "在" "以" "於" "即" "係"
readLines('sheng_dic.txt', encoding = 'UTF-8')
## [1] ""     "經方" "謹上" "滬甯" "京奉" "匯豐" "匯理"
Here we include 10 words in our stopwords list - those we think can be safely filtered out, and we have 6 terms in our custom dictionary so that each of these terms can be segmented as is. It is recommended to use notepad++ to create your custom stopwords lists and dictionaries encoded in UTF-8. Note if you need to use Notepad of Windows to create these text files, it may be easier for R to work with these files if the first rows of each file are left blank - you can see the first elements in my two text files are empty.
# set up and apply the worker and function for segmenting
cutter <- worker(stop_word = 'sheng_stop.txt', user = 'sheng_dic.txt')
seg_x <- function(x) {str_c(cutter[x], collapse = " ")} 
mydata$ltext.seg <- sapply(mydata$ltext, seg_x, USE.NAMES = FALSE)

# view the first few rows
head(mydata, 2)
##                lid
## 1 36-001A-1—001A-2
## 2 36-001B-1—001B-2
##                                                                                                                                                                                                                                                                                                           ltext
## 1                                                              宮保鈞座:新正趨賀兩次,聞藎躬尚未康復,不可以風,未敢驚擾。昨往拜朱邇典。伊面稱久違侍鈞顔,欲於日內晉謁,兼答拜經方。頃接其來函,准於明日下午三句鐘來署拜謁。可否允其接見?謹請示復,以便函告該使。肅此。上叩勛祺,順頌新禧。經方謹上。初九日。
## 2 啟者:頃據民政部公啟稱:於本月十六日下午三點鐘,在部開第一次衛生會會議。請飭派出各員届時蒞會等因。査本部知醫者惟屈道永秋( 去年九月始行札調到部),及承政廳行走學部主事謝天保兩員。屈道在本部月支薪水二百兩,謝主事則於滬甯鐵路掛名,月支洋百元。此兩員在本部無事,可否派其前往會議?請酌示。經方謹上。十二日。
##                                                                                                                                                                                                                                                                                                                                 ltext.seg
## 1                                                                     宮保 鈞座 新 正 趨 賀 兩次 聞 藎 躬 尚未 康復 不 可以 風 未敢 驚擾 昨往 拜 朱邇 典 伊面 稱 久違 侍鈞 顔 欲 日 內 晉謁 兼 答拜 經方 頃接 其 來函 准 明日 下午 三句 鐘來署 拜謁 可否 允其 接見 謹 請示 復 以便 函告 該 使 肅此 上叩 勛 祺 順頌 新禧 經方 謹上 初九 日
## 2 啟者 頃 據 民政部 公 啟稱 本月 十六日 下午 三點鐘 部開 第一次 衛生 會 會議 請 飭 派出 各員届 時 蒞會 等 因 査 本部 知 醫者 惟屈 道 永秋 去年 九月 始行 札 調到 部 及承政廳 行走 學部 主事謝 天保 兩員 屈道 本部 月 支 薪水 二百兩 謝主事則 滬甯 鐵路 掛名 月 支洋 百元 此 兩員 本部 無事 可否 派 其 前往 會議 請 酌 示 經方 謹上 十二日

Create corpus, DFM and feature frequency tab

Now we have segmented texts saved in the variable ltext.seg of mydata. We then use functions corpus() and dfm() to create corpus and DFM from ltext.seg and save them as mycorpus and mydfm.
# create and examine corpus 
mycorpus <- corpus(mydata$ltext.seg)
summary(mycorpus, showmeta = TRUE, 5)
## Corpus consisting of 335 documents, showing 5 documents:
## 
##   Text Types Tokens Sentences
##  text1    73     79         1
##  text2    79     95         1
##  text3   119    148         1
##  text4    62     68         1
##  text5   165    277         1
## 
## Source:  /Users/Guest/Desktop/sheng/* on x86_64 by Guest
## Created: Wed Dec 27 15:14:41 2017
## Notes:
# view texts in the first document of the corpus
texts(mycorpus)[1]
##                                                                                                                                                                                                                                                                 text1 
## "宮保 鈞座 新 正 趨 賀 兩次 聞 藎 躬 尚未 康復 不 可以 風 未敢 驚擾 昨往 拜 朱邇 典 伊面 稱 久違 侍鈞 顔 欲 日 內 晉謁 兼 答拜 經方 頃接 其 來函 准 明日 下午 三句 鐘來署 拜謁 可否 允其 接見 謹 請示 復 以便 函告 該 使 肅此 上叩 勛 祺 順頌 新禧 經方 謹上 初九 日"
# create and examine DFM
mydfm <- dfm(tokens(mydata$ltext.seg, what = "fasterword"))
mydfm 
## Document-feature matrix of: 335 documents, 15,126 features (99.2% sparse).
# top 20 features
topfeatures(mydfm, 20)
##   亦 宮保   已 經方 鈞座 謹上   又   復   稟   不   者 肅頌   呈   均   其 
##  271  265  256  219  208  196  195  187  177  161  160  149  145  143  134 
##   請   再 崇綏   而   寳 
##  131  131  130  125  123
Note in this DFM, those terms included in our stopwords list are gone, and those in our dictionary are segmented as in the text file of the dictionary.
We can then generate a data frame indicating frequency of each features using textstat_frequency().
# tabulate feature frequency
dfmtab <- textstat_frequency(mydfm)
head(dfmtab)
##   feature frequency rank docfreq
## 1      亦       271    1     106
## 2    宮保       265    2     236
## 3      已       256    3     141
## 4    經方       219    4     211
## 5    鈞座       208    5     208
## 6    謹上       196    6     196
Sometimes you only care about, say, longer features/terms, use dfm_select() to choose those meet certain conditions, e.g., terms contain two or more words.
# select 2+ word features 
mydfm2 <- dfm_select(mydfm, min_nchar = 2) 
topfeatures(mydfm2, 20)
##   宮保   經方   鈞座   謹上   肅頌   崇綏   不知   合同   公司   不能 
##    265    219    208    196    149    130     97     90     85     80 
##   閣下   一切   謹悉 外務部   本部   尚未   本日   大人   如何   可以 
##     73     59     58     57     54     53     53     53     53     51
Plot a wordcloud graph from the DFM containing those 2-or-more-word terms. Here we select terms appearing 5 or more times to plot and set 200 as the maximum number of terms to be included.
# plot wordcloud
par(family='Kaiti TC')
textplot_wordcloud(mydfm2, min.freq=5, random.order=FALSE, max.words = 200,
                   rot.per = .25, scale = c(2.8, .5),
                   colors = RColorBrewer::brewer.pal(8,"Dark2"))

Explore more about the data

The function textstat_frequency() can tabulate all feature frequencies like we did above - we can also limit the frequencies to be tabulated and plot these selected features using ggplot() of the ggplot2 package.
# tabulate the top 10 features
textstat_frequency(mydfm2, n=10)
##    feature frequency rank docfreq
## 1     宮保       265    1     236
## 2     經方       219    2     211
## 3     鈞座       208    3     208
## 4     謹上       196    4     196
## 5     肅頌       149    5     149
## 6     崇綏       130    6     130
## 7     不知        97    7      49
## 8     合同        90    8      47
## 9     公司        85    9      45
## 10    不能        80   10      56
# plot freq. by rank of the most frequent 50 features
library(ggplot2)
theme_set(theme_minimal())
textstat_frequency(mydfm2, n = 50) %>% 
  ggplot(aes(x = rank, y = frequency)) +
  geom_point() +
  labs(x = "Frequency rank", y = "Term frequency")

We can also use dfm_weight to create a DFM representing weighted frequencies of the terms, for instance, a DFM with relative feature/term frequencies, i.e., the proportion of the feature counts of total feature counts.
# create dfm with relative term frequencies
dfmpct <- dfm_weight(mydfm2, type = "relfreq") 

# plot relative term frequencies
textstat_frequency(dfmpct, n = 10) %>% 
  ggplot(aes(x = reorder(feature, -rank), y = frequency)) +
  geom_bar(stat = "identity") + coord_flip() + 
  labs(x = "", y = "Relative Term Frequency") +
  theme(text = element_text(family = 'STKaiti'))





In Sum...


In this blog post, we share our way of preparing Chinese texts for computational text analysis, mainly using two R packages - jiebaR and quanteda. We hope this will help our users either to use our Sheng Xuanhuai correspondences collection or to apply this way of processing Chinese texts on your text data. The two packages provide a lot more functions than what we can introduce in this single post, to learn more about the two text analysis packages, start from their documentations at https://qinwenfeng.com/jiebaR/. and http://docs.quanteda.io/index.html.

Wednesday, December 27, 2017

Writing and Local Identity: Literary Women of the Pearl River Delta of Guangdong in the 18th and 19th Centuries

The Library is pleased to organise the following public talk by Prof. Grace S. Fong of McGill University. All are welcome to join! Writin... thumbnail 1 summary
The Library is pleased to organise the following public talk by Prof. Grace S. Fong of McGill University. All are welcome to join!

Writing and Local Identity: Literary Women of the Pearl River Delta of Guangdong in the 18th and 19th Centuries

Speaker
Prof. Grace Fong
Professor of Chinese Literature, Department of East Asian Studies, McGill University
Visiting Professor, School of Chinese, The University of Hong Kong
Abstract of the talk
On the southern margins of the Qing empire, Guangdong has been seen as an evolving site of regional culture, education, and commerce, particularly in the Pearl River Delta counties surrounding the provincial capital Guangzhou in the eighteenth and nineteenth centuries. While anthropologists have uncovered a working-class women’s culture in this region, educated women from the counties of Panyu, Shunde, Xinhui, and Zhongshan were a significant but as yet understudied part of the elite culture.
Using data on Guangdong women in the Ming Qing Women’s Writings database and digital archive (http://digital.library.mcgill.ca/mingqing/) as primary sources for comparison, this paper aims to explore the relationship of these women’s writing to the construction of regional culture and identity before the Western powers had a significant impact through trade and missionary efforts beyond the Canton trading zone. To what extent were literary women part of a regional culture in Guangdong in the eighteenth and nineteenth centuries? Using both biographical and textual data and examining the paratexts, poetic themes and topics, and regional and social networks contained in fifteen individual works by Qing women writers from Guangdong, I will examine how a regional or local culture might have been constructed in these textual productions, and to ask how the components of this regional women’s culture – the hopes and desires, social and cultural activities, reflections and ambitions of these women in the Pearl River Delta show differences from or similarities to their contemporaries, the well-known elite educated women in the cultured Yangzi River Delta. 
Date: 10 Jan, 2018 (Wed)
Time: 4:00 – 5:30 p.m.
Venue: Digital Scholarship Lab, G/F, University Library