xy/gitea
Archived
2
9
Fork 3
Code Issues 7 Pull Requests Projects Releases Wiki Activity

WIP: Federated following #6

Closed
xy wants to merge 35 commits from feature-go-ap-inbox-outbox into feature-activitypub
pull from: feature-go-ap-inbox-outbox
merge into: xy:feature-activitypub
Conversation 2 Commits 35 Files Changed 16 +1028 -47
xy commented 2022-06-12 21:22:55 +00:00
Owner
Copy Link

Description moved to https://github.com/go-gitea/gitea/pull/19981

This pull request implements ActivityPub-federated following so users on a Gitea instance can follow users on any other ActivityPub server (Gitea, Mastodon, PeerTube, Pleroma, etc) and vice versa.

Since #19133 hasn't been merged yet, this PR is currently meant for discussing high-level changes like remote users instead of reviewing the code. (If you want to view the code changes, I made a PR on my personal Gitea repository with that)

Why federated following?

Federated following seems like an odd choice for the first federated behavior for Gitea, but it is actually a good place to start:

  1. Following uses only plain AP (ActivityPub) and doesn't require ForgeFed (which is unsupported by go-ap).
  2. Federated following only requires implementing remote federated users. I say "only" because things like federated pull requests and issues also require handling remote repos which is a whole other tricky issue to resolve.
  3. Federated following requires most of the core ActivityPub components like inboxes and outboxes to work, so this PR already does much of the most difficult work and supporting other activity types should be a lot easier.

Inboxes and outboxes

The first commit implements basic inbox and outbox handling. Since we are not implementing C2S ActivityPub for Gitea, the inbox is not persisted to the database and the outbox reuses the Action table for database storage.

Currently, go-ap does not yet support any of the ForgeFed activity types. As a result, this PR does not implement any type-specific processing (so it currently just returns an empty outbox).

In the future, we will add more ActionTypes for the various AP activity types that we wish to support.

Additional information: #5

Remote users

This PR currently stores remote users in the database using the new LoginType Federated, and stores the full username@instance.com as the username and the ActivityPub IRI as the user's website (which is very hacky and needs to be changed). When viewing a remote user's profile in the web UI, you are redirected to their instance.

Additional information: #2

UI

Currently, this PR only adds the backend code for federated following, so there is no UI (yet) for federated following. From the end user's perspective, that means they will at the moment only be able to follow Gitea accounts using non-Gitea AP software like Mastodon, since there is no UI to follow an AP account remotely from a Gitea instance.

TODO

Description moved to https://github.com/go-gitea/gitea/pull/19981 This pull request implements ActivityPub-federated following so users on a Gitea instance can follow users on any other ActivityPub server (Gitea, Mastodon, PeerTube, Pleroma, etc) and vice versa. Since #19133 hasn't been merged yet, this PR is currently meant for discussing high-level changes like remote users instead of reviewing the code. (If you want to view the code changes, I made [a PR on my personal Gitea repository](https://gitea.com/Ta180m/gitea/pulls/6) with that) ## Why federated following? Federated following seems like an odd choice for the first federated behavior for Gitea, but it is actually a good place to start: 1. Following uses only plain AP (ActivityPub) and doesn't require ForgeFed (which is unsupported by go-ap). 2. Federated following only requires implementing remote federated users. I say "only" because things like federated pull requests and issues also require handling remote repos which is a whole other tricky issue to resolve. 3. Federated following requires most of the core ActivityPub components like inboxes and outboxes to work, so this PR already does much of the most difficult work and supporting other activity types should be a lot easier. ## Inboxes and outboxes The first commit implements basic inbox and outbox handling. Since we are not implementing C2S ActivityPub for Gitea, the inbox is not persisted to the database and the outbox reuses the Action table for database storage. Currently, `go-ap` does not yet support any of the ForgeFed activity types. As a result, this PR does not implement any type-specific processing (so it currently just returns an empty outbox). In the future, we will add more `ActionType`s for the various AP activity types that we wish to support. *Additional information: [#5](https://gitea.com/Ta180m/gitea/issues/5)* ## Remote users This PR currently stores remote users in the database using the new LoginType `Federated`, and stores the full `username@instance.com` as the username and the ActivityPub IRI as the user's website (which is very hacky and needs to be changed). When viewing a remote user's profile in the web UI, you are redirected to their instance. *Additional information: [#2](https://gitea.com/Ta180m/gitea/issues/2)* ## UI Currently, this PR only adds the backend code for federated following, so there is no UI (yet) for federated following. From the end user's perspective, that means they will at the moment only be able to follow Gitea accounts using non-Gitea AP software like Mastodon, since there is no UI to follow an AP account remotely from a Gitea instance. ## TODO - [ ] Fix lint errors - [ ] Write integration tests - [ ] [Implement a federated following UI](https://social.exozy.me/@ta180m/108482276841271853) - [ ] Depends on #19133
xy added 1 commit 2022-06-12 21:22:56 +00:00
xy requested review from dachary 2022-06-12 21:23:07 +00:00
dachary commented 2022-06-12 21:54:55 +00:00
Collaborator
Copy Link

Some actions (see this for instance) have textual content (see ActionCommentIssue or ActionCommentPull) that can be represented using non forgefed vocabulary. This would allow for a first implementation to have tests and a partial non empty content until forgefed is implemented.

What do you think?

Some actions (see [this for instance](https://gitea.hostea.org/dachary?tab=activity&date=2022-05-28)) have textual content (see [ActionCommentIssue](https://github.com/go-gitea/gitea/blob/main/models/action.go#L48) or ActionCommentPull) that can be represented using non forgefed vocabulary. This would allow for a first implementation to have tests and a partial non empty content until forgefed is implemented. What do you think?
xy commented 2022-06-12 22:07:41 +00:00
Author
Owner
Copy Link

Some actions (see this for instance) have textual content (see ActionCommentIssue or ActionCommentPull) that can be represented using non forgefed vocabulary. This would allow for a first implementation to have tests and a partial non empty content until forgefed is implemented.

That sounds like a good idea. We could represent all the actions as AP Notes with the textual content as the note content.

> Some actions (see [this for instance](https://gitea.hostea.org/dachary?tab=activity&date=2022-05-28)) have textual content (see [ActionCommentIssue](https://github.com/go-gitea/gitea/blob/main/models/action.go#L48) or ActionCommentPull) that can be represented using non forgefed vocabulary. This would allow for a first implementation to have tests and a partial non empty content until forgefed is implemented. That sounds like a good idea. We could represent all the actions as AP Notes with the textual content as the note content.
👍 1
xy added 3 commits 2022-06-13 21:24:11 +00:00
xy force-pushed feature-go-ap-inbox-outbox from ceac96941e to d2b4667d89 2022-06-13 21:50:51 +00:00 Compare
xy added 1 commit 2022-06-13 22:20:23 +00:00
xy added 3 commits 2022-06-13 22:55:56 +00:00
xy added 1 commit 2022-06-13 23:21:55 +00:00
xy added 2 commits 2022-06-14 00:30:17 +00:00
xy added 1 commit 2022-06-14 00:31:21 +00:00
xy added 1 commit 2022-06-14 01:52:20 +00:00
xy added 1 commit 2022-06-14 17:40:27 +00:00
xy added 1 commit 2022-06-14 17:58:09 +00:00
xy added 2 commits 2022-06-14 21:31:35 +00:00
xy changed title from Implement POSTing to inbox and GETting outbox by reusing the Action table to Federated following 2022-06-14 21:47:15 +00:00
xy changed title from Federated following to WIP: Federated following 2022-06-14 22:07:25 +00:00
xy added 1 commit 2022-06-15 02:13:13 +00:00
xy added 1 commit 2022-06-15 02:35:48 +00:00
xy added 2 commits 2022-06-15 16:00:26 +00:00
xy added 1 commit 2022-06-15 16:38:42 +00:00
Gusted reviewed 2022-06-15 21:55:19 +00:00
models/db/name.go
@ -18,3 +18,3 @@
// AlphaDashDotPattern characters prohibited in a user name (anything except A-Za-z0-9_.-)
AlphaDashDotPattern = regexp.MustCompile(`[^\w-\.]`)
AlphaDashDotPattern = regexp.MustCompile(`[^\w-\.@]`)
Gusted commented 2022-06-15 21:30:52 +00:00
Collaborator

Is this to work around some validation? Ideally you don't want normal users to have this as a allowed character.

Is this to work around some validation? Ideally you don't want normal users to have this as a allowed character.
xy commented 2022-06-15 22:56:20 +00:00
Author
Owner

This was more of a hack to get federated following to work initially. What we want is for remote users to be able to have a @ in their username, but people shouldn't be able to create local accounts with an @.

This was more of a hack to get federated following to work initially. What we want is for remote users to be able to have a `@` in their username, but people shouldn't be able to create local accounts with an `@`.
modules/activitypub/send.go
@ -0,0 +36,4 @@
err = fmt.Errorf("url IRI fetch [%s] failed with status (%d): %s", iri, resp.StatusCode, resp.Status)
return
}
b, err = io.ReadAll(resp.Body)
Gusted commented 2022-06-15 21:34:53 +00:00
Collaborator

Is it possible to limit the reading of a response body? We still don't want to read a gigabyte of body into memory.

Is it possible to limit the reading of a response body? We still don't want to read a gigabyte of body into memory.
xy commented 2022-06-15 23:02:34 +00:00
Author
Owner

Good idea. I think there's a similar open comment on #19133 about something like this too.

Good idea. I think there's a similar open comment on #19133 about something like this too.
👍 1
xy marked this conversation as resolved
modules/activitypub/send.go Outdated
@ -0,0 +56,4 @@
for _, to := range activity.To {
client, _ := NewClient(user, setting.AppURL+"api/v1/activitypub/user/"+user.Name+"#main-key")
resp, _ := client.Post(body, to.GetID().String())
respBody, _ := io.ReadAll(resp.Body)
Gusted commented 2022-06-15 21:35:41 +00:00
Collaborator

Ditto.

Ditto.
👍 1
xy marked this conversation as resolved
modules/activitypub/user.go
@ -0,0 +16,4 @@
Name: name,
Email: name,
LoginType: auth.Federated,
Website: IRI.String(),
Gusted commented 2022-06-15 21:45:08 +00:00
Collaborator

What about using LoginName for this? It's currently using as storing the user's name from the external login's authenticator(so you can have different name in gitea than on your external login's authenticator). It might even work that you can use Gitea's login feature on a different instance, due to 06372f9fd0/services/auth/signin.go (L52-L85) but that's quite optimistic tbh.

Otherwise we can end up with federation_data table or something alike and store all this stuff on that.

What about using `LoginName` for this? It's currently using as storing the user's name from the external login's authenticator(so you can have different name in gitea than on your external login's authenticator). It might even work that you can use Gitea's login feature on a different instance, due to https://gitea.com/Ta180m/gitea/src/commit/06372f9fd0181919fe7ea038130e76758d2e5182/services/auth/signin.go#L52-L85 but that's quite optimistic tbh. Otherwise we can end up with federation_data table or something alike and store all this stuff on that.
xy commented 2022-06-16 03:17:50 +00:00
Author
Owner

If we don't care about rendering remote users on Gitea, we can store everything inside the existing user table instead of making a new federation_data table since it won't be shown anywhere on the UI.

If we don't care about rendering remote users on Gitea, we can store everything inside the existing user table instead of making a new federation_data table since it won't be shown anywhere on the UI.
Gusted commented 2022-06-16 13:30:41 +00:00
Collaborator

I think rendering remote users will be a feature, especially between Gitea instances. For now there's no harm into storing at login name, we can always migrate it later to a new table.

I think rendering remote users will be a feature, especially between Gitea instances. For now there's no harm into storing at login name, we can always migrate it later to a new table.
xy commented 2022-06-17 16:23:53 +00:00
Author
Owner

Is there really a benefit to rendering remote users? We won't be able to display much about the user anyways since we'll only be able to show the information in their Person actor object. The only benefit I can think of is one-click following.

Is there really a benefit to rendering remote users? We won't be able to display much about the user anyways since we'll only be able to show the information in their Person actor object. The only benefit I can think of is one-click following.
Gusted commented 2022-06-17 16:30:03 +00:00
Collaborator

Well, we can display some information right? Rendering a mastadon user is more tricky, but rendering a user from another Gitea instance would be quite nice and can also show who they are following and their activity, (plus their repositories, but that involves some harder steps).

Well, we can display some information right? Rendering a mastadon user is more tricky, but rendering a user from another Gitea instance would be quite nice and can also show who they are following and their activity, (plus their repositories, but that involves some harder steps).
xy commented 2022-06-17 16:35:20 +00:00
Author
Owner

Rendering a mastadon user is more tricky, but rendering a user from another Gitea instance would be quite nice and can also show who they are following and their activity, (plus their repositories, but that involves some harder steps).

Wouldn't you also be able to see all their information and activity if we didn't render the remote user and instead redirected to their profile on their remote instance?

Also, it's not possible to show a remote Gitea user's repositories unless we make an ActivityPub extension (and add it to ForgeFed maybe) or use the remote instance's Gitea API.

> Rendering a mastadon user is more tricky, but rendering a user from another Gitea instance would be quite nice and can also show who they are following and their activity, (plus their repositories, but that involves some harder steps). Wouldn't you also be able to see all their information and activity if we didn't render the remote user and instead redirected to their profile on their remote instance? Also, it's not possible to show a remote Gitea user's repositories unless we make an ActivityPub extension (and add it to ForgeFed maybe) or use the remote instance's Gitea API.
Gusted commented 2022-06-17 17:04:39 +00:00
Collaborator

Wouldn't you also be able to see all their information and activity if we didn't render the remote user and instead redirected to their profile on their remote instance?

Yes, but will I feel the inconvience to go back and forward between instances to vist some users? Yes. Like on mastodon I can click on any user and see some basic information about it and their recent post. I don't have to leave my instance and instead can have a nice and smooth experience. From a user point of view, being able to comment on issue from another instance, but not able to view a user is weird. I'd prefer to avoid going to another instance as much as possible(getting the information from a instance and then render it on your own instance is much efficient etc. than going to a new instance).

^ Thereby, if I would like to follow a user. I have to proceed to fill in my instance where I just came from?

Don't get me wrong, I understand it's hard to implement this and might be a PITA in the beginning, but I do think it's good to stay on the same instance as much as possible when possible.

Also, it's not possible to show a remote Gitea user's repositories unless we make an ActivityPub extension (and add it to ForgeFed maybe) or use the remote instance's Gitea API.

Yeah, we could still redirect those (for now, or forever if technically impossible).

> Wouldn't you also be able to see all their information and activity if we didn't render the remote user and instead redirected to their profile on their remote instance? Yes, but will I feel the inconvience to go back and forward between instances to vist some users? Yes. Like on mastodon I can click on any user and see some basic information about it and their recent post. I don't have to leave my instance and instead can have a nice and smooth experience. From a user point of view, being able to comment on issue from another instance, but not able to view a user is weird. I'd prefer to avoid going to another instance as much as possible(getting the information from a instance and then render it on your own instance is much efficient etc. than going to a new instance). ^ Thereby, if I would like to follow a user. I have to proceed to fill in my instance where I just came from? Don't get me wrong, I understand it's hard to implement this and might be a PITA in the beginning, but I do think it's good to stay on the same instance as much as possible when possible. > Also, it's not possible to show a remote Gitea user's repositories unless we make an ActivityPub extension (and add it to ForgeFed maybe) or use the remote instance's Gitea API. Yeah, we could still redirect those (for now, or forever if technically impossible).
👍 1
routers/api/v1/activitypub/person.go Outdated
@ -80,0 +86,4 @@
response(ctx, binary)
}
// PersonInbox function
Gusted commented 2022-06-15 21:48:16 +00:00
Collaborator

// PersonInbox handles the incoming data of a person's inbox.

`// PersonInbox handles the incoming data of a person's inbox.`
xy marked this conversation as resolved
routers/api/v1/activitypub/person.go Outdated
@ -80,3 +105,2 @@
var jsonmap map[string]interface{}
err = json.Unmarshal(binary, &jsonmap)
body, err := io.ReadAll(ctx.Req.Body)
Gusted commented 2022-06-15 21:45:38 +00:00
Collaborator

Ditto

Ditto
👍 1
xy marked this conversation as resolved
routers/api/v1/activitypub/person.go Outdated
@ -89,0 +113,4 @@
if activity.Type == ap.FollowType {
activitypub.Follow(ctx, activity)
} else {
log.Warn("ActivityStreams type not supported", activity)
Gusted commented 2022-06-15 21:49:52 +00:00
Collaborator 
		log.Warn("ActivityStreams type not supported: %v", activity)

Also maybe return a bad request status? For a developer working with Gitea, we would like to be verbose when we don't support a feature yet, so they don't have to spend hours debugging why X doesn't work.

```go log.Warn("ActivityStreams type not supported: %v", activity) ``` Also maybe return a bad request status? For a developer working with Gitea, we would like to be verbose when we don't support a feature yet, so they don't have to spend hours debugging why X doesn't work.
👍 1
xy marked this conversation as resolved
routers/api/v1/activitypub/response.go
@ -0,0 +25,4 @@
ctx.Resp.Header().Add("Content-Type", activitypub.ActivityStreamsContentType)
ctx.Resp.WriteHeader(http.StatusOK)
binary, err = json.Marshal(jsonmap)
Gusted commented 2022-06-15 21:53:58 +00:00
Collaborator

It's better to use the ctx.JSON() function here to marshal a JSON into the response. Be aware that you need to ctx.Resp.Header().Set("Content-Type", activitypub.ActivityStreamsContentType) after this LOC.

It's better to use the `ctx.JSON()` function here to marshal a JSON into the response. Be aware that you need to ` ctx.Resp.Header().Set("Content-Type", activitypub.ActivityStreamsContentType)` after this LOC.
xy commented 2022-06-15 23:03:06 +00:00
Author
Owner

That's nicer. I'll also update #19133 to do that as well.

That's nicer. I'll also update #19133 to do that as well.
xy commented 2022-06-16 01:26:04 +00:00
Author
Owner

Actually,

ctx.JSON(http.StatusOK, jsonmap)
ctx.Resp.Header().Set("Content-Type", activitypub.ActivityStreamsContentType)

doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers.

Actually, ``` ctx.JSON(http.StatusOK, jsonmap) ctx.Resp.Header().Set("Content-Type", activitypub.ActivityStreamsContentType) ``` doesn't work because the `ctx.JSON()` function already sends the response out and it's too late to edit the headers.
Gusted commented 2022-06-16 13:27:13 +00:00
Collaborator

Either a new functioncan be created for activity streams JSON or a optional argument to the exisiting function.

Either a new functioncan be created for activity streams JSON or a optional argument to the exisiting function.
Gusted commented 2022-06-18 22:25:29 +00:00
Collaborator 
diff --git a/modules/context/context.go b/modules/context/context.go
index dcc43973c..28d8bc733 100644
--- a/modules/context/context.go
+++ b/modules/context/context.go
@@ -422,8 +422,13 @@ func (ctx *Context) Error(status int, contents ...string) {
 }
 
 // JSON render content as JSON
-func (ctx *Context) JSON(status int, content interface{}) {
-       ctx.Resp.Header().Set("Content-Type", "application/json;charset=utf-8")
+func (ctx *Context) JSON(status int, content interface{}, contentType ...string) {
+       headerContentType := "application/json;charset=utf-8"
+       if len(contentType) > 0 {
+               headerContentType = contentType[0]
+       }
+       ctx.Resp.Header().Set("Content-Type", headerContentType)
+
        ctx.Resp.WriteHeader(status)
        if err := json.NewEncoder(ctx.Resp).Encode(content); err != nil {
                ctx.ServerError("Render JSON failed", err)
```diff diff --git a/modules/context/context.go b/modules/context/context.go index dcc43973c..28d8bc733 100644 --- a/modules/context/context.go +++ b/modules/context/context.go @@ -422,8 +422,13 @@ func (ctx *Context) Error(status int, contents ...string) { } // JSON render content as JSON -func (ctx *Context) JSON(status int, content interface{}) { - ctx.Resp.Header().Set("Content-Type", "application/json;charset=utf-8") +func (ctx *Context) JSON(status int, content interface{}, contentType ...string) { + headerContentType := "application/json;charset=utf-8" + if len(contentType) > 0 { + headerContentType = contentType[0] + } + ctx.Resp.Header().Set("Content-Type", headerContentType) + ctx.Resp.WriteHeader(status) if err := json.NewEncoder(ctx.Resp).Encode(content); err != nil { ctx.ServerError("Render JSON failed", err) ```
👍 2
xy added 1 commit 2022-06-15 22:55:13 +00:00
xy added 1 commit 2022-06-16 17:30:46 +00:00
xy added 1 commit 2022-06-16 17:34:14 +00:00
c67e4189de
make generate-swagger to fix typos 
I didn't know that templates/swagger/v1_json.tmpl was machine-generated... 😭
xy added 3 commits 2022-06-16 21:25:31 +00:00
xy added 1 commit 2022-06-16 21:40:55 +00:00
xy added 1 commit 2022-06-17 16:56:53 +00:00
xy added 1 commit 2022-06-18 18:16:11 +00:00
xy added 3 commits 2022-06-18 18:24:50 +00:00
xy force-pushed feature-go-ap-inbox-outbox from 7e23401a1e to fb685bf05a 2022-06-18 18:28:28 +00:00 Compare
xy added 1 commit 2022-06-18 22:06:33 +00:00
xy added 2 commits 2022-06-18 22:18:12 +00:00
xy closed this pull request 2022-07-11 23:40:42 +00:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No reviewers
dachary
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
4 Participants
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: xy/gitea#6
No description provided.
Delete Branch "feature-go-ap-inbox-outbox"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?