Commit Graph

70 Commits

Author SHA1 Message Date
KN4CK3R b748acf2a0
Fixed emoji alias not parsed in links (#16221)
* Do not skip links.

* Restrict text in links to emojis.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-12-15 23:49:12 +00:00
wxiaoguang 6d4172987e
Fix markdown URL parsing (#17924)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-12-11 19:21:36 +02:00
wxiaoguang 6c49517cbd
Fix issue markdown bugs (#17411)
* Bug fix: render Markdown http://AppURL/org/repo/issues/4?a=1&b=2#comment-123 test to HTML correctly, close #17394
* Bug fix: fix the positions of checkboxes in rendered HTML, close #17395
2021-10-23 21:38:12 +08:00
Eng Zer Jun f2e7d5477f
refactor: move from io/ioutil to io and os package (#17109)
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-09-22 13:38:34 +08:00
zeripath 976db2a8b7
Do not show issue context popup on external issues (#17050)
The issues pop-up context cannot work for external issues - therefore do not show
these.

Fix #17047

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-15 09:45:27 +01:00
zeripath 5fbccad906
Fix NPE in fuzzer (#16680)
The fuzzer found an issue with the issue pattern processor where there is a spurious
path.Clean which does not need to be there. This PR also sets the default AppURL for
the fuzzer too.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-13 00:22:05 +02:00
Josef Fröhle 8d962daed6
cleanup code `issueFullPattern` in modules/markup (#16419)
fix #16415
2021-07-15 22:33:56 +02:00
luzpaz e0296b6a6d
Fix various documentation, user-facing, and source comment typos (#16367)
* Fix various doc, user-facing, and source comment typos

Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
2021-07-08 13:38:13 +02:00
zeripath 32fd11395b
Fix relative links in postprocessed images (#16334)
If a pre-post-processed file contains relative img tags these need to be updated
and joined correctly with the prefix. Finally, the node attributes need to be updated.

Fix #16308

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-07-04 10:26:04 +01:00
6543 65548359cc
Add custom emoji support (#16004) 2021-06-29 16:28:38 +02:00
zeripath d55b5eb0d3
Use html.Parse rather than html.ParseFragment (#16223)
* Use html.Parse rather than html.ParseFragment
  There have been a few issues with html.ParseFragment - just use html.Parse instead.

* Skip document node

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-22 00:12:22 +02:00
zeripath 196593e2e9
More efficiently parse shas for shaPostProcessor (#16101)
* More efficiently parse shas for shaPostProcessor

The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.

This PR provides functions for both backends which are much faster to use.

Fix #16092

* Add ShaExistCache to RenderContext

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-21 00:39:12 +02:00
zeripath 0db1048c3a
Run processors on whole of text (#16155)
There is an inefficiency in the design of our processors which means that Emoji
and other processors run in order n^2 time.

This PR forces the processors to process the entirety of text node before passing
back up. The fundamental inefficiency remains but it should be significantly
ameliorated.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-17 11:35:05 +01:00
KN4CK3R 21cde5c439
Fix data URI scramble (#16098)
* Removed unused method.

* No prefix for data uris.

* Added test to prevent regressions.
2021-06-07 18:55:26 +02:00
silverwind d4f28fd4ad
Fix URL of gitea emoji (#15770)
Fixes regression from #15219
2021-05-07 17:34:33 +02:00
Lunny Xiao 9d99f6ab19
Refactor renders (#15175)
* Refactor renders

* Some performance optimization

* Fix comment

* Transform reader

* Fix csv test

* Fix test

* Fix tests

* Improve optimaziation

* Fix test

* Fix test

* Detect file encoding with reader

* Improve optimaziation

* reduce memory usage

* improve code

* fix build

* Fix test

* Fix for go1.15

* Fix render

* Fix comment

* Fix lint

* Fix test

* Don't use NormalEOF when unnecessary

* revert change on util.go

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* rename function

* Take NormalEOF back

Co-authored-by: zeripath <art27@cantab.net>
2021-04-19 18:25:08 -04:00
zeripath b68eb54f95
Clusterfuzz found another way (#15160)
Clusterfuzz found another way so I found another way to stop it

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-26 01:21:00 +02:00
zeripath a587a28434
Fix another clusterfuzz identified issue (#15096)
* Fix another clusterfuzz identified issue

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-22 02:45:01 +01:00
zeripath c0c052bdbb
another clusterfuzz spotted issue (#15032)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 22:37:11 +01:00
zeripath ed31ddc29a
Fix several render issues (#14986)
* Fix an issue with panics related to attributes
* Wrap goldmark render in a recovery function
* Reduce memory use in render emoji
* Use a pipe for rendering goldmark - still needs more work and a limiter

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-16 00:20:05 +01:00
zeripath aa4f9180e4
Clarify the suffices and prefixes of setting.AppSubURL and setting.AppURL (#12999)
Also removes some unnecessary uses of fmt.Sprintf and adds documentation
strings

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-19 22:36:43 +01:00
zeripath 7ab6c77b41
Remove NULs byte arrays passed to PostProcess (#14587)
PostProcess is supposed to be parsing and handling HTML
fragments, but on fuzzing it appears that there is a weird
issue with NUL elements that could cause a memory address
error in downstream libraries.

The simplest solution is to strip out the weird NULs - they
should not be there in any case and would be stripped out
anyway.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-18 02:32:14 +01:00
zeripath 172229966c
Prevent panic on fuzzer provided string (#14405)
* Prevent panic on fuzzer provided string

The fuzzer has found that providing a <body> tag with an attribute to
PostProcess causes a panic. This PR removes any rendered html or body
tags from the output.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Placate lint

* placate lint again

Signed-off-by: Andrew Thornton <art27@cantab.net>

* minor cleanup

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-01-20 23:10:50 +08:00
Nuno Silva 44ff1d6a1e
Render links for commit hashes followed by comma (#14224)
Regex test cases: https://regex101.com/r/mVbPxM/2/

fixes #14223
2021-01-03 23:11:10 +08:00
a1012112796 34df4e5df5
Add mentionable teams to tributeValues and change team mention rules to gh's style (#13198)
* Add mentionable teams to tributeValues

Signed-off-by: a1012112796 <1012112796@qq.com>

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

* Change team mention rules to gh's style

* use org's avator as team avator in ui

Signed-off-by: a1012112796 <1012112796@qq.com>

* Update modules/markup/html.go

* Update models/issue.go

Co-authored-by: Lauris BH <lauris@nix.lv>

* Update models/issue.go

* fix a small nit and update test code

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
2020-12-21 16:39:28 +01:00
mrsdizzie 3512c7e40f
Fix crash in short link processor (#13839)
Fixes #13819
2020-12-04 03:01:42 +01:00
Jimmy Praet 056b8f5b52
Clickable links in pull request (and issue) titles (#13695)
* clickable links in pull request (and issue) titles #13658

reuses the existing logic to render clickable links in commit messages

* dedicated RenderIssueTitle function

applied patch from @mrsdizzie
2020-12-03 11:50:47 +01:00
kolaente 64133126cd
Update golangci-lint to version 1.31.0 (#13102)
This PR updates golangci-lint to the latest version 1.31.0.

The upgrade introduced a new check for which I've fixed or disabled most cases.

Signed-off-by: kolaente <k@knt.li>
2020-10-11 21:27:20 +01:00
silverwind ee047312a1
Fix emoji replacements, make emoji images consistent (#12567)
- Fix emoji not being replaced in issue title change text
- Make the image attributes consistent, add alt, remove align

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-08-23 21:44:53 -04:00
mrsdizzie 4c1ff57f1a
Update emoji regex (#11584)
When matching emoji, use a regex built from the data we have instead of something generic using unicode ranges. A generic regex can't tell the difference between two separate emoji next to each other or one emoji that is built out of two separate emoji next to each other.

This means that emoji that are next to each other without space in between will be now accurately spanned individually with proper title etc...
2020-05-29 17:08:36 +01:00
mrsdizzie 4ed7d2a2bb
Remove last traces of has-emoji class (#11263)
Now that emojify.js has been removed, get rid of all instances of has-emoji class that was only used for that. Support for rendering shortcodes should remain in all of these places so it should still work the same.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-05-01 20:58:45 +03:00
mrsdizzie 4563eb873d
Support unicode emojis and remove emojify.js (#11032)
* Support unicode emojis and remove emojify.js

This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.

This works in a few ways:

First it adds emoji parsing support into gitea itself. This allows us to

 * Render emojis from valid alias (😄)
 * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
 * Easily allow for custom "emoji"
 * Support all emoji rendering and features without javascript
 * Uses plain unicode and lets the system render in appropriate emoji font
 * Doesn't leave us relying on external sources for updates/fixes/features

That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)

For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.

The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.

I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.

I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.

Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.

Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130

* add new shared function emojiHTML

* don't increase emoji size in issue title

* Update templates/repo/issue/view_content/add_reaction.tmpl

Co-Authored-By: 6543 <6543@obermui.de>

* Support for emoji rendering in various templates

* Render code and review comments as they should be

* Better way to handle mail subjects

* insert unicode from tribute selection

* Add template helper for plain text when needed

* Use existing replace function I forgot about

* Don't include emoji greater than Unicode Version 12

Only include emoji and aliases in JSON

* Update build/generate-emoji.go

* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have

* final updates

* code review

* code review

* hard code gitea custom emoji to match previous behavior

* Update .eslintrc

Co-Authored-By: silverwind <me@silverwind.io>

* disable preempt

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
2020-04-28 15:05:39 -03:00
zeripath 812cfd0ad9
Use markdown frontmatter to provide Table of contents, language and frontmatter rendering (#11047)
* Add control for the rendering of the frontmatter
* Add control to include a TOC
* Add control to set language - allows control of ToC header and CJK glyph choice.

Signed-off-by: Andrew Thornton art27@cantab.net
2020-04-24 14:22:36 +01:00
zeripath 9ad2aa8f4d
Use media links for img in post-process (#10515)
* use media links for img in post-process

* do not visit text of anchors
2020-02-28 01:16:05 -03:00
John Olheiser 7d7ab1eeae Issue/PR Context Popups (#9822)
* Add data-index attribute to issue anchors

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Init JS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add required data to anchor

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Finish popup

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Revert changes to html.go

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Better octicon contexts

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Split out popup function for re-use

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Style changes, test fixes, and cross-reference support

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Prefer em to px

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move label margin to base CSS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move JS to separate file.

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move JS to features and fix module

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Remove query-string and hash

Co-Authored-By: silverwind <me@silverwind.io>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2020-01-19 23:39:21 -05:00
zeripath 27757714d0 Change markdown rendering from blackfriday to goldmark (#9533)
* Move to goldmark

Markdown rendering moved from blackfriday to the goldmark.

Multiple subtle changes required to the goldmark extensions to keep
current rendering and defaults.

Can go further with goldmark linkify and have this work within markdown
rendering making the link processor unnecessary.

Need to think about how to go about allowing extensions - at present it
seems that these would be hard to do without recompilation.

* linter fixes

Co-authored-by: Lauris BH <lauris@nix.lv>
2019-12-31 03:53:28 +02:00
John Olheiser e6ceb6880a Prefix all user-generated IDs in markup (#9477)
* Prefix all user-generated IDs in markup
* Add user-content- to IDs in unit-tests
* fixup markdown_test.go
* update the hrefs for the wiki test
* Add blackfriday extension regex

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-23 22:38:50 +00:00
guillep2k 6a90c7e3dd Alternate syntax for cross references (#9116)
* Add support for local vs. remote xrefs

* Add doc for references

* Docs: fix cases not currently supported

* One more doc fix

* Doc: mentions for teams and orgs

* Change !num ref concept, no change in functionality

* Fix test

* Improve table of issue reference types

* Fix paragraph mark
2019-12-01 15:57:05 +02:00
guillep2k f25fd5c8eb Fix team links in HTML rendering (#9127)
* Fix team links in HTML rendering

* Fix check and lint
2019-11-24 11:34:44 -05:00
guillep2k b15f26b1cf Close/reopen issues by keywords in titles and comments (#8866)
* Add close/reopen from comment functionality

* Fix comment

* Rewrite closing/reopening template

* Check xref permissions, move action to services/pull

* Fix RefIsPull field

* Add xref tests

* Fix xref unique filter

* Only highlight keywords for actionable xrefs

* Fix xref neuter filter

* Fix check return status

* Restart CI
2019-11-18 15:13:07 +02:00
guillep2k cea8ea5ae6 Support inline rendering of CUSTOM_URL_SCHEMES (#8496)
* Support inline rendering of CUSTOM_URL_SCHEMES

* Fix lint

* Add tests

* Fix lint
2019-10-15 02:31:09 +01:00
guillep2k 15809d81f7 Rewrite reference processing code in preparation for opening/closing from comment references (#8261)
* Add a markdown stripper for mentions and xrefs

* Improve comments

* Small code simplification

* Move reference code to modules/references

* Fix typo

* Make MarkdownStripper return [][]byte

* Implement preliminary keywords parsing

* Add FIXME comment

* Fix comment

* make fmt

* Fix permissions check

* Fix text assumptions

* Fix imports

* Fix lint, fmt

* Fix unused import

* Add missing export comment

* Bypass revive on implemented interface

* Move mdstripper into its own package

* Support alphanumeric patterns

* Refactor FindAllMentions

* Move mentions test to references

* Parse mentions from reference package

* Refactor code to implement renderizable references

* Fix typo

* Move patterns and tests to the references package

* Fix nil reference

* Preliminary rendering attempt of closing keywords

* Normalize names, comments, general tidy-up

* Add CSS style for action keywords

* Fix permission for admin and owner

* Fix golangci-lint

* Fix golangci-lint
2019-10-13 23:29:10 +01:00
jaqra 7e17424c7e Make link last commit massages in repository home page and commit tables (#8006)
* Make link last commit massages in repository home page and commit tables

* Use RenderCommitMessageLink instead surround with a

* deleted __debug_bin file

* Exclude email to link from latest commit title

* Exclude email processor from commit table

Co-Authored-By: mrsdizzie <info@mrsdizzie.com>

* Add class parameter to a html element creator functions.
Make links underline dashed that are not commit

* fix tests

* Show dashed underline when also not hovered
2019-09-10 12:03:30 +03:00
Rinat b660a732ae feat: highlight issue references with : (#8101)
* feat: highlight issue references with :

e.g. #1287: my commit msg
e.g. ABC-1234: my commit msg

* ref: update model regex to consistent with issueNumericPattern

* test: check highlight issue with : in commits messages
2019-09-05 20:39:54 -04:00
Tamal Saha 171b359877 Use gitea forked macaron (#7933)
Signed-off-by: Tamal Saha <tamal@appscode.com>
2019-08-23 12:40:29 -04:00
Gary Kim 7eed11e5e9 Check commit message hashes before making links (#7713)
* Check commit message hashes before making links

Previously, when formatting commit messages, anything
that looked like SHA1 hashes was turned into a link
using regex. This meant that certain phrases or numbers
such as `777777` or `deadbeef` could be recognized as a commit
even if the repository has no commit with those hashes.

This change will make it so that anything that looks
like a SHA1 hash using regex will then also be checked
to ensure that there is a commit in the repository
with that hash before making a link.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Use gogit to check if commit exists

This commit modifies the commit hash check
in the render for commit messages to use
gogit for better performance.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Make code cleaner

Signed-off-by: Gary Kim <gary@garykim.dev>

* Use rev-parse to check if commit exists

Signed-off-by: Gary Kim <gary@garykim.dev>

* Add and modify tests for checking hashes in html link rendering

Signed-off-by: Gary Kim <gary@garykim.dev>

* Return error in sha1CurrentPatternProcessor

Co-Authored-By: mrsdizzie <info@mrsdizzie.com>

* Import Gitea log module

Signed-off-by: Gary Kim <gary@garykim.dev>

* Revert "Return error in sha1CurrentPatternProcessor"

This reverts commit 28f561cac46ef7e51aa26aefcbe9aca4671366a6.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Add debug logging to sha1CurrentPatternProcessor

This will log errors by the git command run in
sha1CurrentPatternProcessor if the error is one
that was unexpected.

Signed-off-by: Gary Kim <gary@garykim.dev>
2019-08-14 16:04:55 +08:00
Christian Muehlhaeuser 54d96c79b5 Removed unnecessary conversions (#7557)
No need to convert to the same type.
2019-07-23 19:50:39 +01:00
silverwind b209531959 fix hljs unintenionally highlighting commit links (#7244)
* fix hljs unintenionally highlighting commit links

* fix unit tests
2019-06-18 18:31:31 -04:00
kolaente f9ec2f89f2 Add golangci (#6418) 2019-06-12 15:41:28 -04:00
mrsdizzie 0064535ad2 Fix domain name pattern in email regex (#6739)
Fixes #6735
2019-04-24 21:53:41 -04:00