Knowledge Base Archive
This article is part of the Chinoba Knowledge Base.
Explore Chinoba.org →
Artificial Intelligence Technology Machine Learning Technology Natural Language Processing Clojure
An implementation in Clojure of stopword removal to remove unnecessary words in natural language processing.
In the case of word vectors, a word dictionary is created as follows
( "5"
"①"
"④"
"30"
"②"
"③"
"2"
"⑤"
"⑥"
...)Remove a word using the remove function as follows.
(def ex-stopword
->> raw-word-list
(remove (set (read-string (slurp "data/stopword.txt")))))There is also the following approach.
(defn load-stopwords [filename]
(with-open [r (io/reader filename)]
(set (doall (line-seq r)))))
(def is-stopword (load-stopwords "stopwords/english"))
(def tokens
(map #(remove is-stopword (normalize (tokenize %)))
(get-sentences
"I never saw a Purple Cow.
I never hope to see one.
But I can tell you, anyhow.
I'd rather see than be one.")))
The above code was adapted from Clojure Data Analysis Cookbook 2nd.
The Python implementation is shown below for reference.
words = ['this', 'is', 'a', 'pen']
stop_words = ['is', 'a']
changed_words = [word for word in words if word not in stop_words]
print(changed_words)
Chinoba
Intelligence as Relationship
Research Platform
founded by
Masao Watanabe
AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance
Related Research
This topic is part of the Chinoba Knowledge Base.
コメント